# Xihe: A 3D Vision-based Lighting Estimation Framework for Mobile Augmented Reality

**Authors:** Yiqin Zhao, Tian Guo  
**arXiv:** [2106.15280](https://arxiv.org/abs/2106.15280)  
**Date:** 2021-05-30  
**Summary page:** https://www.summarizepaper.com/en/paper/2106.15280v1/

## Key points

- Achieving spatially-variant photorealistic 3D rendering is highly desirable in mobile augmented reality.
- Advancements in mobile 3D vision technology have improved understanding of physical surroundings.
- Xihe is an edge-assisted framework designed by Yiqin Zhao and Tian Guo for accurate omnidirectional lighting estimation in real-time for mobile AR applications.
- Xihe uses a novel sampling technique to compress raw point cloud input efficiently based on empirical analysis of a recent 3D indoor dataset.
- A tailored GPU pipeline was developed for on-device point cloud processing, and an encoding technique reduces network transmitted bytes.
- An adaptive triggering strategy allows Xihe to skip unnecessary lighting estimations while providing practical ways to provide temporal coherent rendering integration with the mobile AR ecosystem.
- The authors evaluated both the lighting estimation accuracy and time of Xihe using a reference mobile application developed with Xihe's APIs.
- Results show that Xihe takes as fast as 20.67ms per lighting estimation and achieves 9.4% better estimation accuracy than state-of-the-art neural networks.
- Overall, Xihe represents an important step forward in enabling more advanced spatially-variant photorealistic 3D rendering in mobile augmented reality applications by leveraging recent advancements in mobile 3D vision technology.

## Summary

In the world of mobile augmented reality, achieving spatially-variant photorealistic 3D rendering is a highly desirable property. Recently, advancements in mobile 3D vision technology have provided the means to better sense and understand physical surroundings. Yiqin Zhao and Tian Guo have designed an edge-assisted framework called Xihe that provides mobile AR applications with accurate omnidirectional lighting estimation in real-time. The key idea behind Xihe is centered around 3D vision which allows for a more comprehensive understanding of the physical environment. Specifically, they have developed a novel sampling technique that efficiently compresses raw point cloud input generated at the mobile device based on empirical analysis of a recent 3D indoor dataset. To achieve real-time performance goals, Zhao and Guo developed a tailored GPU pipeline for on-device point cloud processing and used an encoding technique that reduces network transmitted bytes. They also presented an adaptive triggering strategy that allows Xihe to skip unnecessary lighting estimations while providing practical ways to provide temporal coherent rendering integration with the mobile AR ecosystem. The authors evaluated both the lighting estimation accuracy and time of Xihe using a reference mobile application developed with Xihe's APIs. Their results show that Xihe takes as fast as 20.67ms per lighting estimation and achieves 9.4% better estimation accuracy than state-of-the-art neural networks. Overall, Xihe represents an important step forward in enabling more advanced spatially-variant photorealistic 3D rendering in mobile augmented reality applications by leveraging recent advancements in mobile 3D vision technology.