Abstract

The goal of Online Domain Adaptation for semantic segmentation is to handle unforeseeable domain changes that occur during deployment, like sudden weather events. However, the high computational costs associated with brute-force adaptation make this paradigm unfeasible for real-world applications. In this paper we propose HAMLET, a Hardware-Aware Modular Least Expensive Training framework for real-time domain adaptation. Our approach includes a hardware-aware back-propagation orchestration agent (HAMT) and a dedicated domain-shift detector that enables active control over when and how the model is adapted (LT). Thanks to these advancements, our approach is capable of performing semantic segmentation while simultaneously adapting at more than 29FPS on a single consumer-grade GPU. Our framework's encouraging accuracy and speed trade-off is demonstrated on OnDA and SHIFT benchmarks through experimental results.

Method overview

Our framework HAMLET is designed to deal with the problem of online domain adaptation with real-time performance through several synergistic strategies. We introduce a Hardware-Aware Modular Training (HAMT) agent able to optimize online a trade-off between model accuracy and adaptation time. HAMT allows us to significantly reduce online training time and GFLOPS. As the second strategy, we introduce a formal geometric model for online domain shifts that enable reliable domain shift detection and domain estimator signals (Adaptive Domain Detection) which can be easily integrated to activate the adaptation process only only when necessary as well as tweaking sensitive training parameters.

Citation

@inproceedings{colomer2023toadapt,
      title = {To Adapt or Not to Adapt? Real-Time Adaptation for Semantic Segmentation},
      author = {Botet Colomer, Marc and 
                Dovesi, Pier Luigi and 
                Panagiotakopoulos, Theodoros and 
                Carvalho, Joao Frederico and 
                H{\"a}renstam-Nielsen, Linus and 
                Azizpour, Hossein and 
                Kjellstr{\"o}m, Hedvig and 
                Cremers, Daniel and
                Poggi, Matteo},
      booktitle = {IEEE International Conference on Computer Vision},
      note = {ICCV},
      year = {2023}
}
              

Acknowledgements

The authors thank Gianluca Villani for the insightful discussion on reward-punishment policies, Leonardo Ravaglia for his expertise on hardware-aware training, and Lorenzo Andraghetti for exceptional technical support throughout the project. Their assistance was invaluable in the completion of this work.