$\text{DC}^2$: Dual-Camera Defocus Control by Learning to Refocus

AI-generated keywords: Defocus Control Dual-Camera Smartphone Cameras Detail Fusion Network Real-World Data

AI-generated Key Points

  • Smartphone cameras have improved in recent years, but still have a limitation with fixed aperture preventing control of depth of field (DoF)
  • Many smartphones now have multiple cameras with different fixed apertures to address this issue
  • Researchers propose $\text{DC}^2$, a system for defocus control that allows for synthetically varying camera aperture, focus distance, and arbitrary defocus effects by fusing information from such a dual-camera system
  • The researchers used the Google Pixel 6 Pro as their camera platform and captured a dataset of 100 focus stacks of diverse scenes to train their model
  • The resulting Detail Fusion Network (DFNet) performs detail fusion on two primary inputs: the reference wide (W) and ultra-wide (UW) images
  • Quantitative and qualitative evaluations on real-world data demonstrate DC2's efficacy where it outperforms state-of-the-art methods on defocus deblurring, bokeh rendering, and image refocus tasks
  • Creative post-capture defocus control enabled by DC2 includes tilt-shift and content based defocus effects
  • DC2 presents an innovative framework for defocus control with dual camera consumer smartphones that bypasses issues related to synthetic data or domain gap due to its training on real data captured with smartphone devices
  • This method benefits from asymmetry in W and UW configurations so it may not perform as well in systems with identical cameras; future work could explore utilizing additional cameras to jointly model both scene depth and defocus control.
Also access our AI generated: Comprehensive summary, Lay summary, Blog-like article; or ask questions about this paper to our AI assistant.

Authors: Hadi Alzayer, Abdullah Abuolaim, Leung Chun Chan, Yang Yang, Ying Chen Lou, Jia-Bin Huang, Abhishek Kar

CVPR 2023. See the project page at https://defocus-control.github.io
License: CC BY 4.0

Abstract: Smartphone cameras today are increasingly approaching the versatility and quality of professional cameras through a combination of hardware and software advancements. However, fixed aperture remains a key limitation, preventing users from controlling the depth of field (DoF) of captured images. At the same time, many smartphones now have multiple cameras with different fixed apertures -- specifically, an ultra-wide camera with wider field of view and deeper DoF and a higher resolution primary camera with shallower DoF. In this work, we propose $\text{DC}^2$, a system for defocus control for synthetically varying camera aperture, focus distance and arbitrary defocus effects by fusing information from such a dual-camera system. Our key insight is to leverage real-world smartphone camera dataset by using image refocus as a proxy task for learning to control defocus. Quantitative and qualitative evaluations on real-world data demonstrate our system's efficacy where we outperform state-of-the-art on defocus deblurring, bokeh rendering, and image refocus. Finally, we demonstrate creative post-capture defocus control enabled by our method, including tilt-shift and content-based defocus effects.

Submitted to arXiv on 06 Apr. 2023

Ask questions about this paper to our AI assistant

You can also chat with multiple papers at once here.

AI assistant instructions?

Results of the summarizing process for the arXiv paper: 2304.03285v1

Smartphone cameras have come a long way in recent years, with hardware and software advancements allowing them to approach the versatility and quality of professional cameras. However, one key limitation remains: fixed aperture prevents users from controlling the depth of field (DoF) of captured images. To address this issue, many smartphones now have multiple cameras with different fixed apertures - an ultra-wide camera with a wider field of view and deeper DoF, and a higher resolution primary camera with shallower DoF. In this context, researchers propose $\text{DC}^2$, a system for defocus control that allows for synthetically varying camera aperture, focus distance, and arbitrary defocus effects by fusing information from such a dual-camera system. The key insight behind their approach is to leverage real-world smartphone camera datasets by using image refocus as a proxy task for learning to control defocus. To train their model, the researchers used the Google Pixel 6 Pro as their camera platform and captured a dataset of 100 focus stacks of diverse scenes. For each scene, they swept the focus plane for the wide camera and simultaneously captured a frame from the ultra-wide camera. They then used optical-flow-based warping to align the ultra-wide frame with the wide frame. The resulting Detail Fusion Network (DFNet) performs detail fusion on two primary inputs: the reference wide (W) and ultra-wide (UW) images. DFNet has two refinement paths: W refinement path ($\Phi_{W}$), which treats W as a base image; and UW refinement path ($\Phi_{UW}$), which serves as a guide for missing high-frequency details. Quantitative and qualitative evaluations on real-world data demonstrate DC2's efficacy where it outperforms state-of-the-art methods on defocus deblurring, bokeh rendering, and image refocus tasks. Additionally, creative post-capture defocus control enabled by DC2 includes tilt-shift and content based defocus effects. Overall, DC2 presents an innovative framework for defocus control with dual camera consumer smartphones that bypasses issues related to synthetic data or domain gap due to its training on real data captured with smartphone devices. However, it should be noted that this method benefits from asymmetry in W and UW configurations so it may not perform as well in systems with identical cameras; thus future work could explore utilizing additional cameras to jointly model both scene depth and defocus control.
Created on 09 Apr. 2023

Assess the quality of the AI-generated content by voting

Score: 0

Why do we need votes?

Votes are used to determine whether we need to re-run our summarizing tools. If the count reaches -10, our tools can be restarted.

Similar papers summarized with our AI tools

Navigate through even more similar papers through a

tree representation

Look for similar papers (in beta version)

By clicking on the button above, our algorithm will scan all papers in our database to find the closest based on the contents of the full papers and not just on metadata. Please note that it only works for papers that we have generated summaries for and you can rerun it from time to time to get a more accurate result while our database grows.

Disclaimer: The AI-based summarization tool and virtual assistant provided on this website may not always provide accurate and complete summaries or responses. We encourage you to carefully review and evaluate the generated content to ensure its quality and relevance to your needs.