Enhancing Autonomous Driving with Multi-Modal Data Fusion

Researchers at Queen Mary University of London are developing an innovative machine-learning framework for multi-modal data fusion to improve emergency responses in autonomous vehicle driving. The research is expected to…

Home  Research Updates    Enhancing Autonomous Driving with Multi-Modal Data Fusion

Researchers at Queen Mary University of London are developing an innovative machine-learning framework for multi-modal data fusion to improve emergency responses in autonomous vehicle driving.

The research is expected to positively impact time-sensitive applications such as autonomous driving systems, smart city infrastructure and intelligent transportation while fostering collaboration across various fields. It has many potential real-world impacts such as improved emergency response systems, traffic management and reduction of accidents, better delivery robots and Advanced Driver Assistance Systems (ADAS), and improved efficiency of robotic systems used in manufacturing, healthcare and other industries.

This project addresses the limitations of traditional fusion algorithms, which often struggle with the high coupling of feature extractors for different modalities, making flexible expansion difficult. Additionally, the new framework reduces the training cost associated with obtaining large amounts of labelled data and lightens communication overhead. The primary goal of this project is to create an efficient and convenient machine-learning framework for multi-modal data fusion, which is crucial for emergency responses in autonomous vehicle driving under wireless communications. The framework aims to decouple the joint optimisation problem across different modalities, simplifying model training and ensuring effective representation of the latent space encompassing all modalities.

Research findings

The research has yielded several significant findings:

  • Effective Use of Unlabelled Data: Variational Autoencoders (VAEs) can effectively utilise unlabelled data for feature extraction. Once sufficiently trained, the encoder acts as a feature extractor, requiring only a small amount of labelled data for supervised learning tasks such as classification or prediction.
  • Decoupling Joint Optimisation: By introducing two independence assumptions, the researchers have managed to decouple the joint optimisation problem across different modalities. This approach simplifies model training, as the VAEs for each modality do not need to participate in joint gradient descent during training.
  • Auxiliary Optimisation Objectives: By sampling from the outputs of the encoder and decoder, the two assumptions can be transformed into auxiliary optimisation objectives by minimising Kullback–Leibler divergence and mutual information. These auxiliary tasks act as regularisation terms that guide the model’s iterative process, ensuring that the assumptions hold.

The research promises to significantly enhance real-time performance for autonomous driving by integrating information from multiple modalities, such as cameras, LiDARs, inertial measurement units, and GNSS receivers. The study provides an edge computing solution that brings feature extractors close to the data source, reducing latency and communication pressure. Additionally, the modular approach simplifies model development and maintenance, allowing for flexible replacement or modification of modality-specific modules.

A further extension grant application for testing on a practical overseas testbed has been submitted, highlighting the project’s potential for real-world application and further development.

Looking ahead, the team plans to explore how different modalities are represented in the latent space and improve the efficiency of the auxiliary tasks. They aim to extend the framework to more complex multi-modal time-series analysis tasks, enhancing its adaptability for diverse applications.

The research is a part of TITAN Lighthouse Project 1: Network Intelligence and End-to-End Integration. Lighthouse Project 1 (LP1) focuses on network intelligence and end-to-end integration. The work’s outcomes will help drive improvements in network performance, efficiency, and the ability to provide seamless and reliable services.

If you have any questions about the research discussed in this article, please contact titan-enquiries@eng.cam.ac.uk.