# Seeing Through Fog Without Seeing Fog: Deep Sensor Fusion in the Absence of Labeled Training Data

**Authors:** Mario Bijelic, Fahim Mannan, Tobias Gruber, Werner Ritter, Klaus Dietmayer, Felix Heide  
**arXiv:** [1902.08913](https://arxiv.org/abs/1902.08913)  
**Date:** 2019-02-24  
**Summary page:** https://www.summarizepaper.com/en/paper/1902.08913v1/

## Key points

- Fusion of color and lidar data is crucial for object detection in autonomous vehicles
- Existing fusion methods often fail to effectively utilize redundant and complementary information in adverse weather and imaging conditions
- Challenging "edge-case" scenarios are not well-represented in available datasets
- Current fusion architectures are not designed to handle severe asymmetric distortions
- Authors propose a deep fusion architecture for robust fusion in foggy and snowy conditions without large labeled training data specific to these scenarios
- Introduce a real-time single-shot model that adaptively fuses features based on temporal coherence of distortions
- Proposed method validated through simulations and real-world driving scenarios with unseen conditions
- Model trained on clean data but demonstrates ability to handle foggy and snowy environments effectively
- Deep fusion architecture leverages temporal coherence of distortions to overcome challenges posed by adverse weather conditions
- Offers potential advancements in autonomous vehicle technology.

## Summary

The fusion of color and lidar data is crucial for object detection in autonomous vehicles, as their decision-making relies heavily on these inputs. However, existing methods for fusion often fail to effectively utilize redundant and complementary information in adverse weather and imaging conditions, where the sensory streams can be asymmetrically distorted. These challenging "edge-case" scenarios are not well-represented in available datasets, and current fusion architectures are not designed to handle severe asymmetric distortions. To address this issue, the authors propose a deep fusion architecture that enables robust fusion in foggy and snowy conditions, even without access to large labeled training data specific to these scenarios. Departing from traditional proposal-level fusion approaches, they introduce a real-time single-shot model that adaptively fuses features based on the temporal coherence of the distortions. The proposed method is validated through simulations and real-world driving scenarios with unseen conditions. The model is trained on clean data but demonstrates its ability to handle foggy and snowy environments effectively. By leveraging the temporal coherence of distortions, the deep fusion architecture overcomes challenges posed by adverse weather conditions and provides reliable object detection capabilities for autonomous vehicles. Overall, this research presents an innovative approach to sensor fusion in challenging weather conditions, offering potential advancements in autonomous vehicle technology.