Interactive3D: Create What You Want by Interactive 3D Generation

Shaocong Dong^* · Lihe Ding^* · Zhanpeng Huang · Zibin Wang · Tianfan Xue · Dan Xu

Paper | Project Page | Video

---

NEWS

• 🔥 Interactive3D got accepted by CVPR24.
• Code will be released soon.

Abstract

Interactive3D is an innovative framework for interactive 3D generation that grants users precise control over the generative process through extensive 3D interaction capabilities. Interactive3D is constructed in two cascading stages, utilizing distinct 3D representations. The first stage employs Gaussian Splatting for direct user interaction, allowing modifications and guidance of the generative direction at any intermediate step through (i) Adding and Removing components, (ii) Deformable and Rigid Dragging, (iii) Geometric Transformations, and (iv) Semantic Editing. Subsequently, the Gaussian splats are transformed into InstantNGP. We introduce a novel (v) Interactive Hash Refinement module to further add details and extract the geometry in the second stage.

Acknowledgement

Interactive3D is built on many amazing research works, thanks a lot to all the authors for sharing! Thank Yiyuan for the valuable discussion and paper refinement.

• gsgen and instant-ngp
• threestudio

Citation

If the paper and the code are helpful for your research, please kindly cite:

@article{dong2024interactive3d,
  title={Interactive3D: Create What You Want by Interactive 3D Generation},
  author={Dong, Shaocong and Ding, Lihe and Huang, Zhanpeng and Wang, Zibin and Xue, Tianfan and Xu, Dan},
  journal={arXiv preprint arXiv:2404.16510},
  year={2024}
}