PrEditor3D: Fast and Precise 3D Shape Editing

Ziya Erkoç1    Can Gümeli1    Chaoyang Wang2    Matthias Niessner1    Angela Dai1    Peter Wonka2,3    Hsin-Ying Lee2    Peiye Zhuang2
1TUM   2Snap Inc   3KAUST
Work partially done while Ziya and Can were interning at Snap
arXiv Video GitHub

PrEditor3D quickly edits a 3D shape with a given text prompt and a mask.

Abstract

We propose a training-free approach to 3D editing that enables the editing of a single shape within a few minutes. The edited 3D mesh aligns well with the prompts, and remains identical for regions that are not intended to be altered. To this end, we first project the 3D object onto 4-view images and perform synchronized multi-view image editing along with user-guided text prompts and user-provided rough masks. However, the targeted regions to be edited are ambiguous due to projection from 3D to 2D. To ensure precise editing only in intended regions, we develop a 3D segmentation pipeline that detects edited areas in 3D space, followed by a merging algorithm to seamlessly integrate edited 3D regions with the original input. Extensive experiments demonstrate the superiority of our method over previous approaches, enabling fast, high-quality editing while preserving unintended regions.

Video

Precise Editing Results

Prompt & Mask Input Shape Edited Shape
truck carrying clay pizza
viking with a red pepper mustache wearing a helmet
robotic arm holding a Oreo cookie pizza over a coffee mug
hogwarts castle with main clock tower in the middle
chicken cat with a tail in a racing car

Diverse Editing Results

With the same prompt and different seeds, we can generate diverse results.
By adjusting the prompt, we can control different aspects of editing such as appearance and mood.
happy ginger cat ... sad ginger cat ... ginger cat with sunglasses ...

Ablation Results

Our merging algorithm allows us to keep original parts of the shape fixed. In the example below, naively doing multi-view editing without any merging changes important properties of the car as well. With our mergin operation, we can recover the car back. However, that can lead to disjoint parts which we resolve by doing additional averaging around the new part.
Input Shape w/o Merging w/o Average Merging Ours

BibTeX

@misc{erkoc2024preditor3d,
    title={PrEditor3D: Fast and Precise 3D Shape Editing}, 
    author={Ziya Erkoç and Can Gümeli and Chaoyang Wang and Matthias Nießner and Angela Dai and Peter Wonka and Hsin-Ying Lee and Peiye Zhuang},
    year={2024},
    eprint={2412.06592},
    archivePrefix={arXiv},
    primaryClass={cs.CV},
    url={https://arxiv.org/abs/2412.06592}, 
}