Self-Supervised Correspondence Estimation via Multiview Registration
Auteurs : Mohamed El Banani, Ignacio Rocco, David Novotny, Andrea Vedaldi, Natalia Neverova, Justin Johnson, Benjamin Graham
Résumé scientifique
Video provides us with the spatio-temporal consistency needed for visual learning. Recent approaches have utilized this signal to learn correspondence estimation from close-by frame pairs. However, by only relying on close-by frame pairs, those approaches miss out on the richer long-range consistency between distant overlapping frames. To address this, we propose a self-supervised approach for correspondence estimation that learns from multiview consistency in short RGB-D video sequences. Our approach combines pairwise correspondence estimation and registration with a novel SE(3) transformation synchronization algorithm. Our key insight is that self-supervised multiview registration allows us to obtain correspondences over longer time frames; increasing both the diversity and difficulty of sampled pairs. We evaluate our approach on indoor scenes for correspondence estimation and RGB-D pointcloud registration and find that we perform on-par with supervised approaches.
Les articles autour de lui, par date et par proximité
Plus haut, c’est plus proche de cet article ; à gauche, c’est plus ancien. Les traits relient les articles proches entre eux : les groupes de traits sont des sous-thèmes. Sélectionnez un point pour voir pourquoi il est là. Retour au résumé de cet article
Les articles autour de celui-ci
Classés selon la proximité de leur titre et de leur résumé avec ceux de cet article. Sélectionnez-en pour les lire ensemble.
Les articles les plus proches
-
Self-Supervised Cross-View Correspondence with Predictive Cycle ConsistencyAlan Baade, Changan Chen · 2025 · Computer Vision and Pattern Recognition · cité 4 fois · Pas encore résumé
Résumé scientifique
Learning self-supervised visual correspondence is a long-studied task fundamental to visual understanding and human perception. However, existing correspondence methods largely focus on small image transformations, such as object tracking in high-framerate videos or learning pixel-to-pixel mappings between images with high view overlap. This severely limits their application in dynamic multi-view settings such as robot imitation learning. In this work, we introduce Predictive Cycle Consistency for learning object correspondence between extremely disjoint views of a scene without paired segmentation data. Our technique bootstraps object correspondence pseudolabels from raw image segmentations…
-
SCENES: Subpixel Correspondence EstimationWith Epipolar SupervisionDominik A. Kloepfer, João F. Henriques, Dylan Campbell · 2024 · International Conference on 3D Vision · cité 2 fois · Pas encore résumé
Résumé scientifique
Extracting point correspondences from two or more views of a scene is a fundamental computer vision problem with particular importance for relative camera pose estimation and structure-from-motion. Existing local feature matching approaches, trained with correspondence supervision on large-scale datasets, obtain highly-accurate matches on the test sets. However, they do not generalise well to new datasets with different characteristics to those they were trained on, unlike classic feature extractors. Instead, they require finetuning, which assumes that ground-truth correspondences or ground-truth camera poses and 3D structure are available. We relax this assumption by removing the requiremen…
-
Learning Correspondence From the Cycle-Consistency of TimeX. Wang, A. Jabri, Alexei A. Efros · 2019 · Computer Vision and Pattern Recognition · cité 558 fois · Pas encore résumé
Résumé scientifique
We introduce a self-supervised method for learning visual correspondence from unlabeled video. The main idea is to use cycle-consistency in time as free supervisory signal for learning visual representations from scratch. At training time, our model learns a feature map representation to be useful for performing cycle-consistent tracking. At test time, we use the acquired representation to find nearest neighbors across space and time. We demonstrate the generalizability of the representation -- without finetuning -- across a range of visual correspondence tasks, including video object segmentation, keypoint tracking, and optical flow. Our approach outperforms previous self-supervised methods…
-
Discriminative Correspondence Estimation for Unsupervised RGB-D Point Cloud RegistrationChenbo Yan, Ming-Tao Feng, Zi-Jie Wu et al. · 2025 · IEEE transactions on circuits and systems for video technology (Print) · cité 22 fois · Pas encore résumé
Résumé scientifique
Point cloud registration is a fundamental task for estimating the rigid transformation matrix between two point clouds, and is regarded as a prerequisite for downstream vision tasks. Recent works have sought to address the registration problem using the obtainable RGB-D sequence, rather than relying solely on point clouds, which may not always be available. However, most existing unsupervised RGB-D point cloud registration works struggle to obtain fine-grained, robust, discriminative correspondences due to the simple concatenation of multimodal features and the increase in vector dimensions. These methods typically follow a common paradigm: extracting features from the input data, estimating…
-
Leveraging Cycle-Consistent Anchor Points for Self-Supervised RGB-D RegistrationSiddharth Tourani, Jayaram Reddy, Sarvesh Thakur et al. · 2024 · IEEE International Conference on Robotics and Automation · cité 3 fois · Pas encore résumé
Résumé scientifique
With the rise in consumer depth cameras, a wealth of unlabeled RGB-D data has become available. This prompts the question of how to utilize this data for geometric reasoning of scenes. While many RGB-D registration methods rely on geometric and feature-based similarity, we take a different approach. We use cycle-consistent keypoints as salient points to enforce spatial coherence constraints during matching, improving correspondence accuracy. Additionally, we introduce a novel pose block that combines a GRU recurrent unit with transformation synchronization, blending historical and multi-view data. Our approach surpasses previous self-supervised registration methods on ScanNet and 3DMatch, ev…
-
Learning Multiview 3D Point Cloud RegistrationZan Gojcic, Cai-Fa Zhou, J. Wegner et al. · 2020 · Computer Vision and Pattern Recognition · cité 192 fois · Pas encore résumé
Résumé scientifique
We present a novel, end-to-end learnable, multiview 3D point cloud registration algorithm. Registration of multiple scans typically follows a two-stage pipeline: the initial pairwise alignment and the globally consistent refinement. The former is often ambiguous due to the low overlap of neighboring point clouds, symmetries and repetitive scene parts. Therefore, the latter global refinement aims at establishing the cyclic consistency across multiple scans and helps in resolving the ambiguous cases. In this paper we propose, to the best of our knowledge, the first end-to-end algorithm for joint learning of both parts of this two-stage problem. Experimental evaluation on well accepted benchmar…
-
Space-Time Correspondence as a Contrastive Random WalkA. Jabri, Andrew Owens, Alexei A. Efros · 2020 · Neural Information Processing Systems · cité 370 fois · Pas encore résumé
Résumé scientifique
This paper proposes a simple self-supervised approach for learning representations for visual correspondence from raw video. We cast correspondence as link prediction in a space-time graph constructed from a video. In this graph, the nodes are patches sampled from each frame, and nodes adjacent in time can share a directed edge. We learn a node embedding in which pairwise similarity defines transition probabilities of a random walk. Prediction of long-range correspondence is efficiently computed as a walk along this graph. The embedding learns to guide the walk by placing high probability along paths of correspondence. Targets are formed without supervision, by cycle-consistency: we train th…
-
UnsupervisedR&R: Unsupervised Point Cloud Registration via Differentiable RenderingMohamed El Banani, Luya Gao, Justin Johnson · 2021 · Computer Vision and Pattern Recognition · cité 76 fois · Pas encore résumé
Résumé scientifique
Aligning partial views of a scene into a single whole is essential to understanding one’s environment and is a key component of numerous robotics tasks such as SLAM and SfM. Recent approaches have proposed end-to-end systems that can outperform traditional methods by leveraging pose supervision. However, with the rising prevalence of cameras with depth sensors, we can expect a new stream of raw RGB-D data without the annotations needed for supervision. We propose UnsupervisedR&R: an end-to-end unsupervised approach to learning point cloud registration from raw RGB-D video. The key idea is to leverage differentiable alignment and rendering to enforce photometric and geometric consistency betw…
Les travaux antérieurs sur lesquels il s’appuie
-
Bootstrap Your Own CorrespondencesMohamed El Banani, Justin Johnson · 2021 · IEEE International Conference on Computer Vision · cité 50 fois · Pas encore résumé
Résumé scientifique
Geometric feature extraction is a crucial component of point cloud registration pipelines. Recent work has demonstrated how supervised learning can be leveraged to learn better and more compact 3D features. However, those approaches’ reliance on ground-truth annotation limits their scalability. We propose BYOC: a self-supervised approach that learns visual and geometric features from RGB-D video without relying on ground-truth pose or correspondence. Our key observation is that randomly-initialized CNNs readily provide us with good correspondences; allowing us to bootstrap the learning of both visual and geometric features. Our approach combines classic ideas from point cloud registration wi…
-
Learning to Find Good CorrespondencesK. M. Yi, E. Trulls, Yuki Ono et al. · 2017 · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · cité 540 fois · Pas encore résumé
Résumé scientifique
We develop a deep architecture to learn to find good correspondences for wide-baseline stereo. Given a set of putative sparse matches and the camera intrinsics, we train our network in an end-to-end fashion to label the correspondences as inliers or outliers, while simultaneously using them to recover the relative pose, as encoded by the essential matrix. Our architecture is based on a multi-layer perceptron operating on pixel coordinates rather than directly on the image, and is thus simple and small. We introduce a novel normalization technique, called Context Normalization, which allows us to process each data point separately while embedding global information in it, and also makes the n…
-
Learning Transformation SynchronizationXiangru Huang, Zhenxiao Liang, Xiao-Wei Zhou et al. · 2019 · Computer Vision and Pattern Recognition · cité 64 fois · Pas encore résumé
Résumé scientifique
Reconstructing the 3D model of a physical object typically requires us to align the depth scans obtained from different camera poses into the same coordinate system. Solutions to this global alignment problem usually proceed in two steps. The first step estimates relative transformations between pairs of scans using an off-the-shelf technique. Due to limited information presented between pairs of scans, the resulting relative transformations are generally noisy. The second step then jointly optimizes the relative transformations among all input depth scans. A natural constraint used in this step is the cycle-consistency constraint, which allows us to prune incorrect relative transformations …
-
SuperPoint: Self-Supervised Interest Point Detection and DescriptionDaniel DeTone, Tomasz Malisiewicz, A. Rabinovich · 2017 · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · cité 3 867 fois · Pas encore résumé
Résumé scientifique
This paper presents a self-supervised framework for training interest point detectors and descriptors suitable for a large number of multiple-view geometry problems in computer vision. As opposed to patch-based neural networks, our fully-convolutional model operates on full-sized images and jointly computes pixel-level interest point locations and associated descriptors in one forward pass. We introduce Homographic Adaptation, a multi-scale, multi-homography approach for boosting interest point detection repeatability and performing cross-domain adaptation (e.g., synthetic-to-real). Our model, when trained on the MS-COCO generic image dataset using Homographic Adaptation, is able to repeated…
-
Digging Into Self-Supervised Learning of Feature DescriptorsIaroslav Melekhov, Zakaria Laskar, Xiaotian Li et al. · 2021 · International Conference on 3D Vision · cité 9 fois · Pas encore résumé
Résumé scientifique
Fully-supervised CNN-based approaches for learning local image descriptors have shown remarkable results in a wide range of geometric tasks. However, most of them require per-pixel ground-truth keypoint correspondence data which is difficult to acquire at scale. To address this challenge, recent weakly-and self-supervised methods can learn feature descriptors from relative camera poses or using only synthetic rigid transformations such as homographies. In this work, we focus on understanding the limitations of existing self-supervised approaches and propose a set of improvements that combined lead to powerful feature descriptors. We show that increasing the search space from in-pair to in-ba…
Les articles qui le citent
-
Robust Multiview Point Cloud Registration with Reliable Pose Graph Initialization and History ReweightingHai-Ping Wang, Yuan Liu, Zhen Dong et al. · 2023 · Computer Vision and Pattern Recognition · cité 50 fois · Pas encore résumé
Résumé scientifique
In this paper, we present a new method for the multi-view registration of point cloud. Previous multiview registration methods rely on exhaustive pairwise registration to construct a densely-connected pose graph and apply Iteratively Reweighted Least Square (IRLS) on the pose graph to compute the scan poses. However, constructing a densely-connected graph is time-consuming and contains lots of outlier edges, which makes the subsequent IRLS struggle to find correct poses. To address the above problems, we first propose to use a neural network to estimate the overlap between scan pairs, which enables us to construct a sparse but reliable pose graph. Then, we design a novel history reweighting …
-
Doduo: Learning Dense Visual Correspondence from Unsupervised Semantic-Aware FlowZhen-Yu Jiang, Hanwen Jiang, Yuke Zhu · 2023 · IEEE International Conference on Robotics and Automation · cité 10 fois · Pas encore résumé
Résumé scientifique
Dense visual correspondence plays a vital role in robotic perception. This work focuses on establishing the dense correspondence between a pair of images that captures dynamic scenes undergoing substantial transformations. We introduce Doduo to learn general dense visual correspondence from in-the-wild images and videos without ground truth supervision. Given a pair of images, it estimates the dense flow field encoding the displacement of each pixel in one image to its corresponding pixel in the other image. Doduo uses flow-based warping to acquire supervisory signals for the training. Incorporating semantic priors with self-supervised flow training, Doduo produces accurate dense corresponde…
-
Geometry Depth Consistency in RGBD Relative Pose EstimationSourav Kumar, Chiang-Heng Chien, Benjamin B. Kimia · 2024 · arXiv.org · preprint, non relu par les pairs · Pas encore résumé
Résumé scientifique
Relative pose estimation for RGBD cameras is crucial in a number of applications. Previous approaches either rely on the RGB aspect of the images to estimate pose thus not fully making use of depth in the estimation process or estimate pose from the 3D cloud of points that each image produces, thus not making full use of RGB information. This paper shows that if one pair of correspondences is hypothesized from the RGB-based ranked-ordered correspondence list, then the space of remaining correspondences is restricted to corresponding pairs of curves nested around the hypothesized correspondence, implicitly capturing depth consistency. This simple Geometric Depth Constraint (GDC) significantly…
-
Unsupervised Rgb-D Point Cloud Registration for Scenes With Low Overlap and Photometric InconsistencyYejun Shou, Hao-Cheng Wang, Lingfeng Shen et al. · 2025 · IEEE International Conference on Computer Vision · cité 3 fois · Pas encore résumé
-
PF3plat: Pose-Free Feed-Forward 3D Gaussian SplattingSung-Jin Hong, Jaewoo Jung, Heeseong Shin et al. · 2024 · arXiv.org · preprint, non relu par les pairs · cité 75 fois · Pas encore résumé
Résumé scientifique
We consider the problem of novel view synthesis from unposed images in a single feed-forward. Our framework capitalizes on fast speed, scalability, and high-quality 3D reconstruction and view synthesis capabilities of 3DGS, where we further extend it to offer a practical solution that relaxes common assumptions such as dense image views, accurate camera poses, and substantial image overlaps. We achieve this through identifying and addressing unique challenges arising from the use of pixel-aligned 3DGS: misaligned 3D Gaussians across different views induce noisy or sparse gradients that destabilize training and hinder convergence, especially when above assumptions are not met. To mitigate thi…
Aller plus loin
Comment ces articles sont-ils trouvés ?
Nous comparons le titre et le résumé de cet article avec ceux de tous les articles résumés sur ce site, et avec les articles que Semantic Scholar recommande, ceux qu’il cite et ceux qui le citent (pour un article très cité, ses citations les plus récentes et, depuis OpenAlex, les plus citées). Parmi les travaux antérieurs et les suites, les articles qui s’utilisent en profondeur, une méthode qu’il utilise et les suites très citées remontent un peu. Les niveaux (très proche, proche, apparenté) viennent de la similarité des textes, pas d’un jugement de qualité ; * signale un article sans résumé, comparé par son titre. La phrase sous un article est écrite par une IA à partir des deux résumés : vérifiez-la dans les articles. Un article jugé hors sujet par trois lecteurs quitte la liste.
Mis à jour le 5 novembre 2026.