AI summary · arXiv: 2212.03236v1 · cs.CV
Self-Supervised Correspondence Estimation via Multiview Registration
The essentials
- The paper addresses the issue of learning correspondence estimation from video sequences
- Previous approaches focus on spatio-temporal consistency from close-by frame pairs, but fail to capture long-range consistency between distant overlapping frames
- The authors propose a self-supervised approach that leverages multiview consistency in short RGB-D video sequences
- The method combines pairwise correspondence estimation and registration using a novel SE(3) transformation synchronization algorithm
- Self-supervised multiview registration allows for obtaining correspondences over longer time frames, increasing diversity and difficulty of sampled pairs
- Experiments were conducted on indoor scenes for correspondence estimation and RGB-D point cloud registration
- Results show that the proposed approach performs on par with supervised approaches, demonstrating its effectiveness in learning accurate correspondences.
Key points written by AI: they may contain errors, check them against the paper.
Ask questions about this paper to our AI assistant
How to chat with this paper? Illustrated guide
You can also chat with multiple papers at once here.
Keep this conversation — a free account saves your questions and answers, your collections of papers and your alerts.
Continue with Google Create a free accountTools for this paper
Advanced tools Premium
For researchers who follow a field closely: citation alerts, journal-club slides, version comparison and Zotero export. Discover Premium
Papers around this one
See them on a mapRanked by how close their title and abstract are to this paper’s. Select some to read them together.
Closest papers
-
Self-Supervised Cross-View Correspondence with Predictive Cycle ConsistencyAlan Baade, Changan Chen · 2025 · Computer Vision and Pattern Recognition · cited 4 times · Not summarized yet
Abstract
Learning self-supervised visual correspondence is a long-studied task fundamental to visual understanding and human perception. However, existing correspondence methods largely focus on small image transformations, such as object tracking in high-framerate videos or learning pixel-to-pixel mappings between images with high view overlap. This severely limits their application in dynamic multi-view settings such as robot imitation learning. In this work, we introduce Predictive Cycle Consistency for learning object correspondence between extremely disjoint views of a scene without paired segmentation data. Our technique bootstraps object correspondence pseudolabels from raw image segmentations…
-
SCENES: Subpixel Correspondence EstimationWith Epipolar SupervisionDominik A. Kloepfer, João F. Henriques, Dylan Campbell · 2024 · International Conference on 3D Vision · cited 2 times · Not summarized yet
Abstract
Extracting point correspondences from two or more views of a scene is a fundamental computer vision problem with particular importance for relative camera pose estimation and structure-from-motion. Existing local feature matching approaches, trained with correspondence supervision on large-scale datasets, obtain highly-accurate matches on the test sets. However, they do not generalise well to new datasets with different characteristics to those they were trained on, unlike classic feature extractors. Instead, they require finetuning, which assumes that ground-truth correspondences or ground-truth camera poses and 3D structure are available. We relax this assumption by removing the requiremen…
-
Learning Correspondence From the Cycle-Consistency of TimeX. Wang, A. Jabri, Alexei A. Efros · 2019 · Computer Vision and Pattern Recognition · cited 558 times · Not summarized yet
Abstract
We introduce a self-supervised method for learning visual correspondence from unlabeled video. The main idea is to use cycle-consistency in time as free supervisory signal for learning visual representations from scratch. At training time, our model learns a feature map representation to be useful for performing cycle-consistent tracking. At test time, we use the acquired representation to find nearest neighbors across space and time. We demonstrate the generalizability of the representation -- without finetuning -- across a range of visual correspondence tasks, including video object segmentation, keypoint tracking, and optical flow. Our approach outperforms previous self-supervised methods…
-
Discriminative Correspondence Estimation for Unsupervised RGB-D Point Cloud RegistrationChenbo Yan, Ming-Tao Feng, Zi-Jie Wu et al. · 2025 · IEEE transactions on circuits and systems for video technology (Print) · cited 22 times · Not summarized yet
Abstract
Point cloud registration is a fundamental task for estimating the rigid transformation matrix between two point clouds, and is regarded as a prerequisite for downstream vision tasks. Recent works have sought to address the registration problem using the obtainable RGB-D sequence, rather than relying solely on point clouds, which may not always be available. However, most existing unsupervised RGB-D point cloud registration works struggle to obtain fine-grained, robust, discriminative correspondences due to the simple concatenation of multimodal features and the increase in vector dimensions. These methods typically follow a common paradigm: extracting features from the input data, estimating…
-
Leveraging Cycle-Consistent Anchor Points for Self-Supervised RGB-D RegistrationSiddharth Tourani, Jayaram Reddy, Sarvesh Thakur et al. · 2024 · IEEE International Conference on Robotics and Automation · cited 3 times · Not summarized yet
Abstract
With the rise in consumer depth cameras, a wealth of unlabeled RGB-D data has become available. This prompts the question of how to utilize this data for geometric reasoning of scenes. While many RGB-D registration methods rely on geometric and feature-based similarity, we take a different approach. We use cycle-consistent keypoints as salient points to enforce spatial coherence constraints during matching, improving correspondence accuracy. Additionally, we introduce a novel pose block that combines a GRU recurrent unit with transformation synchronization, blending historical and multi-view data. Our approach surpasses previous self-supervised registration methods on ScanNet and 3DMatch, ev…
-
Learning Multiview 3D Point Cloud RegistrationZan Gojcic, Cai-Fa Zhou, J. Wegner et al. · 2020 · Computer Vision and Pattern Recognition · cited 192 times · Not summarized yet
Abstract
We present a novel, end-to-end learnable, multiview 3D point cloud registration algorithm. Registration of multiple scans typically follows a two-stage pipeline: the initial pairwise alignment and the globally consistent refinement. The former is often ambiguous due to the low overlap of neighboring point clouds, symmetries and repetitive scene parts. Therefore, the latter global refinement aims at establishing the cyclic consistency across multiple scans and helps in resolving the ambiguous cases. In this paper we propose, to the best of our knowledge, the first end-to-end algorithm for joint learning of both parts of this two-stage problem. Experimental evaluation on well accepted benchmar…
-
Space-Time Correspondence as a Contrastive Random WalkA. Jabri, Andrew Owens, Alexei A. Efros · 2020 · Neural Information Processing Systems · cited 370 times · Not summarized yet
Abstract
This paper proposes a simple self-supervised approach for learning representations for visual correspondence from raw video. We cast correspondence as link prediction in a space-time graph constructed from a video. In this graph, the nodes are patches sampled from each frame, and nodes adjacent in time can share a directed edge. We learn a node embedding in which pairwise similarity defines transition probabilities of a random walk. Prediction of long-range correspondence is efficiently computed as a walk along this graph. The embedding learns to guide the walk by placing high probability along paths of correspondence. Targets are formed without supervision, by cycle-consistency: we train th…
-
UnsupervisedR&R: Unsupervised Point Cloud Registration via Differentiable RenderingMohamed El Banani, Luya Gao, Justin Johnson · 2021 · Computer Vision and Pattern Recognition · cited 76 times · Not summarized yet
Abstract
Aligning partial views of a scene into a single whole is essential to understanding one’s environment and is a key component of numerous robotics tasks such as SLAM and SfM. Recent approaches have proposed end-to-end systems that can outperform traditional methods by leveraging pose supervision. However, with the rising prevalence of cameras with depth sensors, we can expect a new stream of raw RGB-D data without the annotations needed for supervision. We propose UnsupervisedR&R: an end-to-end unsupervised approach to learning point cloud registration from raw RGB-D video. The key idea is to leverage differentiable alignment and rendering to enforce photometric and geometric consistency betw…
Prior work it builds on
-
Bootstrap Your Own CorrespondencesMohamed El Banani, Justin Johnson · 2021 · IEEE International Conference on Computer Vision · cited 50 times · Not summarized yet
Abstract
Geometric feature extraction is a crucial component of point cloud registration pipelines. Recent work has demonstrated how supervised learning can be leveraged to learn better and more compact 3D features. However, those approaches’ reliance on ground-truth annotation limits their scalability. We propose BYOC: a self-supervised approach that learns visual and geometric features from RGB-D video without relying on ground-truth pose or correspondence. Our key observation is that randomly-initialized CNNs readily provide us with good correspondences; allowing us to bootstrap the learning of both visual and geometric features. Our approach combines classic ideas from point cloud registration wi…
-
Learning to Find Good CorrespondencesK. M. Yi, E. Trulls, Yuki Ono et al. · 2017 · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · cited 540 times · Not summarized yet
Abstract
We develop a deep architecture to learn to find good correspondences for wide-baseline stereo. Given a set of putative sparse matches and the camera intrinsics, we train our network in an end-to-end fashion to label the correspondences as inliers or outliers, while simultaneously using them to recover the relative pose, as encoded by the essential matrix. Our architecture is based on a multi-layer perceptron operating on pixel coordinates rather than directly on the image, and is thus simple and small. We introduce a novel normalization technique, called Context Normalization, which allows us to process each data point separately while embedding global information in it, and also makes the n…
-
Learning Transformation SynchronizationXiangru Huang, Zhenxiao Liang, Xiao-Wei Zhou et al. · 2019 · Computer Vision and Pattern Recognition · cited 64 times · Not summarized yet
Abstract
Reconstructing the 3D model of a physical object typically requires us to align the depth scans obtained from different camera poses into the same coordinate system. Solutions to this global alignment problem usually proceed in two steps. The first step estimates relative transformations between pairs of scans using an off-the-shelf technique. Due to limited information presented between pairs of scans, the resulting relative transformations are generally noisy. The second step then jointly optimizes the relative transformations among all input depth scans. A natural constraint used in this step is the cycle-consistency constraint, which allows us to prune incorrect relative transformations …
-
SuperPoint: Self-Supervised Interest Point Detection and DescriptionDaniel DeTone, Tomasz Malisiewicz, A. Rabinovich · 2017 · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · cited 3,867 times · Not summarized yet
Abstract
This paper presents a self-supervised framework for training interest point detectors and descriptors suitable for a large number of multiple-view geometry problems in computer vision. As opposed to patch-based neural networks, our fully-convolutional model operates on full-sized images and jointly computes pixel-level interest point locations and associated descriptors in one forward pass. We introduce Homographic Adaptation, a multi-scale, multi-homography approach for boosting interest point detection repeatability and performing cross-domain adaptation (e.g., synthetic-to-real). Our model, when trained on the MS-COCO generic image dataset using Homographic Adaptation, is able to repeated…
-
Digging Into Self-Supervised Learning of Feature DescriptorsIaroslav Melekhov, Zakaria Laskar, Xiaotian Li et al. · 2021 · International Conference on 3D Vision · cited 9 times · Not summarized yet
Abstract
Fully-supervised CNN-based approaches for learning local image descriptors have shown remarkable results in a wide range of geometric tasks. However, most of them require per-pixel ground-truth keypoint correspondence data which is difficult to acquire at scale. To address this challenge, recent weakly-and self-supervised methods can learn feature descriptors from relative camera poses or using only synthetic rigid transformations such as homographies. In this work, we focus on understanding the limitations of existing self-supervised approaches and propose a set of improvements that combined lead to powerful feature descriptors. We show that increasing the search space from in-pair to in-ba…
Papers that cite it
-
Robust Multiview Point Cloud Registration with Reliable Pose Graph Initialization and History ReweightingHai-Ping Wang, Yuan Liu, Zhen Dong et al. · 2023 · Computer Vision and Pattern Recognition · cited 50 times · Not summarized yet
Abstract
In this paper, we present a new method for the multi-view registration of point cloud. Previous multiview registration methods rely on exhaustive pairwise registration to construct a densely-connected pose graph and apply Iteratively Reweighted Least Square (IRLS) on the pose graph to compute the scan poses. However, constructing a densely-connected graph is time-consuming and contains lots of outlier edges, which makes the subsequent IRLS struggle to find correct poses. To address the above problems, we first propose to use a neural network to estimate the overlap between scan pairs, which enables us to construct a sparse but reliable pose graph. Then, we design a novel history reweighting …
-
Doduo: Learning Dense Visual Correspondence from Unsupervised Semantic-Aware FlowZhen-Yu Jiang, Hanwen Jiang, Yuke Zhu · 2023 · IEEE International Conference on Robotics and Automation · cited 10 times · Not summarized yet
Abstract
Dense visual correspondence plays a vital role in robotic perception. This work focuses on establishing the dense correspondence between a pair of images that captures dynamic scenes undergoing substantial transformations. We introduce Doduo to learn general dense visual correspondence from in-the-wild images and videos without ground truth supervision. Given a pair of images, it estimates the dense flow field encoding the displacement of each pixel in one image to its corresponding pixel in the other image. Doduo uses flow-based warping to acquire supervisory signals for the training. Incorporating semantic priors with self-supervised flow training, Doduo produces accurate dense corresponde…
-
Geometry Depth Consistency in RGBD Relative Pose EstimationSourav Kumar, Chiang-Heng Chien, Benjamin B. Kimia · 2024 · arXiv.org · preprint, not peer reviewed · Not summarized yet
Abstract
Relative pose estimation for RGBD cameras is crucial in a number of applications. Previous approaches either rely on the RGB aspect of the images to estimate pose thus not fully making use of depth in the estimation process or estimate pose from the 3D cloud of points that each image produces, thus not making full use of RGB information. This paper shows that if one pair of correspondences is hypothesized from the RGB-based ranked-ordered correspondence list, then the space of remaining correspondences is restricted to corresponding pairs of curves nested around the hypothesized correspondence, implicitly capturing depth consistency. This simple Geometric Depth Constraint (GDC) significantly…
-
Unsupervised Rgb-D Point Cloud Registration for Scenes With Low Overlap and Photometric InconsistencyYejun Shou, Hao-Cheng Wang, Lingfeng Shen et al. · 2025 · IEEE International Conference on Computer Vision · cited 3 times · Not summarized yet
-
PF3plat: Pose-Free Feed-Forward 3D Gaussian SplattingSung-Jin Hong, Jaewoo Jung, Heeseong Shin et al. · 2024 · arXiv.org · preprint, not peer reviewed · cited 75 times · Not summarized yet
Abstract
We consider the problem of novel view synthesis from unposed images in a single feed-forward. Our framework capitalizes on fast speed, scalability, and high-quality 3D reconstruction and view synthesis capabilities of 3DGS, where we further extend it to offer a practical solution that relaxes common assumptions such as dense image views, accurate camera poses, and substantial image overlaps. We achieve this through identifying and addressing unique challenges arising from the use of pixel-aligned 3DGS: misaligned 3D Gaussians across different views induce noisy or sparse gradients that destabilize training and hinder convergence, especially when above assumptions are not met. To mitigate thi…
Go further
How are these papers found?
We compare the title and abstract of this paper with those of every paper summarized on this site, and with the papers Semantic Scholar recommends, those it cites and those citing it (for a much-cited paper, its most recent citations and, from OpenAlex, the most cited ones). Among the prior work and the follow-ups, the papers that use each other in depth, a method it uses and the much-cited follow-ups come a little higher. The levels (very close, close, related) come from the similarity of the texts, not from a judgment of quality; * marks a paper without an abstract, compared by its title. The sentence under a paper is written by an AI from the two abstracts: check it in the papers. A paper marked not relevant by three readers leaves the list.
Updated on 5 November 2026.
The previous summary was created more than a year ago and can be re-run (if necessary) by clicking on the Run button below.
Disclaimer: The AI-based summarization tool and virtual assistant provided on this website may not always provide accurate and complete summaries or responses. We encourage you to carefully review and evaluate the generated content to ensure its quality and relevance to your needs.