Robust traffic analysis in smart mobility

While Fraunhofer IVI scientists are developing an algorithm for data fusion and processing of radar data, Fraunhofer IAIS experts are working on the development of the corresponding AI model for recognition and localization (detection) of traffic participants on infrared images. They are also supporting the Fraunhofer IVI in the development of the underlying algorithm. The Fraunhofer institutes FHR and IIS not only support the implementation of the project and advise on data campaigns, but are also responsible for the proof of concept of the AI model for the classification of traffic participants.

Flexibility and robustness through combination of radar and infrared data

In contrast to conventional forms of AI models for the detection of people and vehicles, the Fraunhofer approach is based on the combination of two different modalities for data acquisition: the fusion of radar and infrared data creates a robust analysis system for which even temporary occlusions of traffic participants do not cause a problem - the data acquisition is thus very reliable even in difficult lighting conditions and can thus be used regardless of the daytime, in real time and in almost any weather. Object detection is also supplemented by a tracking method that provides additional robustness and enables the analysis of traffic participants in image sequences and 3D. The collection and analysis of sensitive data from public spaces is carried out in compliance with data protection regulations.

What is the Fraunhofer technology all about?

Traffic is recorded using various sensors. Classical clustering methods are used for detection in the radar data. The evaluation of the infrared data is based on a deep learning approach with a corresponding Convolutional Neural Network (CNN). The fusion of both detection sources is realized, among others, by using a Hungarian algorithm. The choice of sensors is not standardized, but in combination is very robust to a wide range of problems. Furthermore, the sensors also complement each other in localization, so that a 3D position of traffic participants can be determined.

How are privacy compliant recordings and evaluations possible?

Both sensors are not suitable for identifying individuals, which reduces the risk of data privacy issues in the data collection phase. In addition, neither images nor videos are recorded to ensure that the privacy of traffic participants is protected.

Why does the technology work even under difficult weather conditions and in the dark?

Neither the infrared nor the radar sensor unit is directly dependent on the sun or other illumination.
Simplified, most objects in the infrared range emit in some proportion to their temperature. This self-emissivity makes objects, as well as people and (moving) motorized vehicles, visible regardless of illumination. In addition, some types of fog are less opaque in the infrared range than in the visible range, so you can see through them.

In contrast, radar is an active sensing system. This means that it emits its own signal and detects the response from the environment - regardless of the prevailing light conditions. In addition, unlike the commonly used visual system, the radar system is robust to impairments such as temporary occlusions of traffic participants.

The Fraunhofer technology in practice

Tailored to specific use cases, recorded traffic data can be evaluated. This allows the prediction of traffic flows or real-time communication with autonomous vehicles or driver assistance systems. Furthermore, real-time communication can be an important tool in traffic management - such as for traffic light switching - or can be used to identify free routes for emergency services.