Inpaint Anything: Segment Anything Meets Image Inpainting

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

1. Close Not relevant

   Generative Image Inpainting with Contextual Attention

   Jiahui Yu, Zhe Lin, Jimei Yang et al. · 2018 · Computer Vision and Pattern Recognition · AI summary available

   Abstract

   Recent deep learning based approaches have shown promising results on image inpainting for the challenging task of filling in large missing regions in an image. These methods can generate visually plausible image structures and textures, but often create distorted structures or blurry textures inconsistent with surrounding areas. This is mainly due to ineffectiveness of convolutional neural networks in explicitly borrowing or copying information from distant spatial locations. On the other hand, traditional texture and patch synthesis approaches are particularly suitable when it needs to borrow textures from the surrounding regions. Motivated by these observations, we propose a new deep gene…

2. Close Not relevant

   Image Inpainting for Irregular Holes Using Partial Convolutions

   Guilin Liu, Fitsum A. Reda, Kevin J. Shih et al. · 2018 · Computer Vision and Pattern Recognition · AI summary available

   Abstract

   Existing deep learning based image inpainting methods use a standard convolutional network over the corrupted image, using convolutional filter responses conditioned on both valid pixels as well as the substitute values in the masked holes (typically the mean value). This often leads to artifacts such as color discrepancy and blurriness. Post-processing is usually used to reduce such artifacts, but are expensive and may fail. We propose the use of partial convolutions, where the convolution is masked and renormalized to be conditioned on only valid pixels. We further include a mechanism to automatically generate an updated mask for the next layer as part of the forward pass. Our model outper…

3. Close Not relevant

   VIP: Versatile Image Outpainting Empowered by Multimodal Large Language Model

   Jinze Yang, Haoran Wang, Zining Zhu et al. · 2024 · Computer Vision and Pattern Recognition · AI summary available

   Abstract

   In this paper, we focus on resolving the problem of image outpainting, which aims to extrapolate the surrounding parts given the center contents of an image. Although recent works have achieved promising performance, the lack of versatility and customization hinders their practical applications in broader scenarios. Therefore, this work presents a novel image outpainting framework that is capable of customizing the results according to the requirement of users. First of all, we take advantage of a Multimodal Large Language Model (MLLM) that automatically extracts and organizes the corresponding textual descriptions of the masked and unmasked part of a given image. Accordingly, the obtained t…

4. Close Not relevant

   ACE++: Instruction-Based Image Creation and Editing via Context-Aware Content Filling

   Chaojie Mao, Jingfeng Zhang, Yulin Pan et al. · 2025 · Computer Vision and Pattern Recognition · AI summary available

   Abstract

   We report ACE++, an instruction-based diffusion framework that tackles various image generation and editing tasks. Inspired by the input format for the inpainting task proposed by FLUX.1-Fill-dev, we improve the Long-context Condition Unit (LCU) introduced in ACE and extend this input paradigm to any editing and generation tasks. To take full advantage of image generative priors, we develop a two-stage training scheme to minimize the efforts of finetuning powerful text-to-image diffusion models like FLUX.1-dev. In the first stage, we pre-train the model using task data with the 0-ref tasks from the text-to-image model. There are many models in the community based on the post-training of text…

5. Related Not relevant

   A Simple Latent Diffusion Approach for Panoptic Segmentation and Mask Inpainting

   Wouter Van Gansbeke, Bert De Brabandere · 2024 · Computer Vision and Pattern Recognition · AI summary available

   Abstract

   Panoptic and instance segmentation networks are often trained with specialized object detection modules, complex loss functions, and ad-hoc post-processing steps to handle the permutation-invariance of the instance masks. This work builds upon Stable Diffusion and proposes a latent diffusion approach for panoptic segmentation, resulting in a simple architecture which omits these complexities. Our training process consists of two steps: (1) training a shallow autoencoder to project the segmentation masks to latent space; (2) training a diffusion model to allow image-conditioned sampling in latent space. The use of a generative model unlocks the exploration of mask completion or inpainting, wh…

6. Related Not relevant

   MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing

   Kai Zhang, Lingbo Mo, Wenhu Chen et al. · 2023 · Computer Vision and Pattern Recognition · AI summary available

   Abstract

   Text-guided image editing is widely needed in daily life, ranging from personal use to professional applications such as Photoshop. However, existing methods are either zero-shot or trained on an automatically synthesized dataset, which contains a high volume of noise. Thus, they still require lots of manual tuning to produce desirable outcomes in practice. To address this issue, we introduce MagicBrush (https://osu-nlp-group.github.io/MagicBrush/), the first large-scale, manually annotated dataset for instruction-guided real image editing that covers diverse scenarios: single-turn, multi-turn, mask-provided, and mask-free editing. MagicBrush comprises over 10K manually annotated triples (so…

7. Related Not relevant

   Segment Anything

   Alexander Kirillov, Eric Mintun, Nikhila Ravi et al. · 2023 · Computer Vision and Pattern Recognition · AI summary available

   Abstract

   We introduce the Segment Anything (SA) project: a new task, model, and dataset for image segmentation. Using our efficient model in a data collection loop, we built the largest segmentation dataset to date (by far), with over 1 billion masks on 11M licensed and privacy respecting images. The model is designed and trained to be promptable, so it can transfer zero-shot to new image distributions and tasks. We evaluate its capabilities on numerous tasks and find that its zero-shot performance is impressive -- often competitive with or even superior to prior fully supervised results. We are releasing the Segment Anything Model (SAM) and corresponding dataset (SA-1B) of 1B masks and 11M images at…

8. Related Not relevant

   Continuous-Multiple Image Outpainting in One-Step via Positional Query and A Diffusion-based Approach

   Shaofeng Zhang, Jinfa Huang, Qiang Zhou et al. · 2024 · Computer Vision and Pattern Recognition · AI summary available

   Abstract

   Image outpainting aims to generate the content of an input sub-image beyond its original boundaries. It is an important task in content generation yet remains an open problem for generative models. This paper pushes the technical frontier of image outpainting in two directions that have not been resolved in literature: 1) outpainting with arbitrary and continuous multiples (without restriction), and 2) outpainting in a single step (even for large expansion multiples). Moreover, we develop a method that does not depend on a pre-trained backbone network, which is in contrast commonly required by the previous SOTA outpainting methods. The arbitrary multiple outpainting is achieved by utilizing …