Humans as Light Bulbs: 3D Human Reconstruction from Thermal Reflection

AI-generated keywords: Thermal Reflections

AI-generated Key Points

  • The paper presents a novel approach to reconstructing the 3D position and pose of a human using thermal reflections on everyday objects.
  • The authors exploit the fact that the human body emits long-wave infrared light, which has a larger wavelength than visible light, causing many surfaces in typical scenes to act as infrared mirrors with strong specular reflections.
  • By analyzing these thermal reflections onto objects, they can locate a person's position and reconstruct their pose, even if they are not visible to a normal camera.
  • The authors propose an analysis-by-synthesis framework that jointly models the objects, people, and their thermal reflections.
  • They evaluate their reconstruction by comparing the 2D keypoints and 3D skeleton estimated from synchronized images captured by a calibrated third camera.
  • Their quantitative experiments and qualitative visualizations show the effectiveness of their technical approach as well as design decisions.
  • Thermal cameras are powerful tools for studying human activities in daily environments extending computer vision systems' ability to function more robustly even under extreme light conditions.
  • Integrating thermal cameras with modern computer vision models will bring out many downstream applications in robotics, graphics, and 3D perception.
Also access our AI generated: Comprehensive summary, Lay summary, Blog-like article; or ask questions about this paper to our AI assistant.

Authors: Ruoshi Liu, Carl Vondrick

Website: https://thermal.cs.columbia.edu/
License: CC BY 4.0

Abstract: The relatively hot temperature of the human body causes people to turn into long-wave infrared light sources. Since this emitted light has a larger wavelength than visible light, many surfaces in typical scenes act as infrared mirrors with strong specular reflections. We exploit the thermal reflections of a person onto objects in order to locate their position and reconstruct their pose, even if they are not visible to a normal camera. We propose an analysis-by-synthesis framework that jointly models the objects, people, and their thermal reflections, which allows us to combine generative models with differentiable rendering of reflections. Quantitative and qualitative experiments show our approach works in highly challenging cases, such as with curved mirrors or when the person is completely unseen by a normal camera.

Submitted to arXiv on 02 May. 2023

Ask questions about this paper to our AI assistant

You can also chat with multiple papers at once here.

AI assistant instructions?

Results of the summarizing process for the arXiv paper: 2305.01652v1

This paper presents a novel approach to reconstructing the 3D position and pose of a human using thermal reflections on everyday objects. The authors exploit the fact that the human body emits long-wave infrared light, which has a larger wavelength than visible light, causing many surfaces in typical scenes to act as infrared mirrors with strong specular reflections. By analyzing these thermal reflections onto objects, they can locate a person's position and reconstruct their pose, even if they are not visible to a normal camera. The authors propose an analysis-by-synthesis framework that jointly models the objects, people, and their thermal reflections. This allows them to combine generative models with differentiable rendering of reflections. They evaluate their reconstruction by comparing the 2D keypoints and 3D skeleton estimated from synchronized images captured by a calibrated third camera. They compare their results to 200 randomly sampled 2D human keypoints and 3D skeletons from the HumanEva dataset. Their quantitative experiments and qualitative visualizations show the effectiveness of their technical approach as well as design decisions. Particularly, they believe their findings regarding differentiable rendering of reflections on implicit surfaces will provide insights to other computer vision researchers working with reflections. The primary contribution of this paper is a method to use thermal reflection of the human body on everyday objects to infer its location in a scene and its 3D structure. Section two provides an overview of related work for 3D reconstruction and differentiable rendering while section three formulates an integrated generative model of humans and objects in a scene before discussing how to perform differentiable rendering of reflection which can be inverted to reconstruct the 3D scene. Section four analyzes the capabilities of this approach in real-world scenarios. The authors believe that thermal cameras are powerful tools for studying human activities in daily environments extending computer vision systems' ability to function more robustly even under extreme light conditions. They conclude that integrating thermal cameras with modern computer vision models will bring out many downstream applications in robotics, graphics, and 3D perception.
Created on 03 May. 2023

Assess the quality of the AI-generated content by voting

Score: 0

Why do we need votes?

Votes are used to determine whether we need to re-run our summarizing tools. If the count reaches -10, our tools can be restarted.

Similar papers summarized with our AI tools

Navigate through even more similar papers through a

tree representation

Look for similar papers (in beta version)

By clicking on the button above, our algorithm will scan all papers in our database to find the closest based on the contents of the full papers and not just on metadata. Please note that it only works for papers that we have generated summaries for and you can rerun it from time to time to get a more accurate result while our database grows.

Disclaimer: The AI-based summarization tool and virtual assistant provided on this website may not always provide accurate and complete summaries or responses. We encourage you to carefully review and evaluate the generated content to ensure its quality and relevance to your needs.