Tetrahedron Splatting for 3D Generation

Chun Gu1, Zeyu Yang1, Zijie Pan1, Xiatian Zhu2, Li Zhang1
1 Fudan University, 2 University of Surrey

Paper Code



Abstract


3D representation is essential to the significant advance of 3D generation with 2D diffusion priors. As a flexible representation, NeRF has been first adopted for 3D representation. With density-based volumetric rendering, it however suffers both intensive computational overhead and inaccurate mesh extraction. Using a signed distance field and Marching Tetrahedra, DMTet allows for precise mesh extraction and real-time rendering but is limited in handling large topological changes in meshes, leading to optimization challenges. Alternatively, 3D Gaussian Splatting (3DGS) is favored in both training and rendering efficiency while falling short in mesh extraction. In this work, we introduce a novel 3D representation, Tetrahedron Splatting (TeT-Splatting), that supports easy convergence during optimization, precise mesh extraction, and real-time rendering simultaneously. This is achieved by integrating surface-based volumetric rendering within a structured tetrahedral grid while preserving the desired ability of precise mesh extraction, and a tile-based differentiable tetrahedron rasterizer. Furthermore, we incorporate eikonal and normal consistency regularization terms for the signed distance field to improve generation quality and stability. Critically, our representation can be trained without mesh extraction, making the optimization process easier to converge. Our TeT-Splatting can be readily integrated in existing 3D generation pipelines, along with polygonal mesh for texture optimization. Extensive experiments show that our TeT-Splatting strikes a superior tradeoff among convergence speed, render efficiency, and mesh quality as compared to previous alternatives under varying 3D generation settings.



Video Demo



Pipeline





Gallery Results

"an amigurumi bulldozer"
"an astronaut standing on the surface of mars"
"an erupting volcano"
"an old car overgrown by vines and weeds"
"an opulent couch from the palace of Versailles"
"an origami crane"
"an origami motorcycle"
"an ornate silver gravy boat sitting on a patterned tablecloth"
"an unstable rock cairn in the middle of a stream"
"a 3d model of an adorable cottage with a thatched roof"
"a baby bunny sitting on top of a stack of pancakes"
"a bald eagle carved out of wood"
"a bear dancing ballet"
"a bear dressed in medieval armor"
"a beautifully carved wooden knight chess piece"
"a beautiful dress made out of garbage bags"
"a blue tulip"
"a brightly colored mushroom growing on a log"
"a cake covered in colorful frosting with a slice being taken out"
"a cake in the shape of a train"
"a candelabra with many candles on a red velvet tablecloth"
"a capybara wearing a top hat"
"a car made out of cheese"
"a cauldron full of gold coins"
"a ceramic lion"
"a chimpanzee looking through a telescope"
"a completely destroyed car"
"a crocodile playing a drum set"
"a delicious croissant"
"a delicious hamburger"
"A DMC Delorean car"
"a DSLR photo of Mont Saint-Michel"
"a ficus planted in a pot"
"a fresh cinnamon roll covered in glaze"
"a giant worm emerging from the sand in the middle of the desert"
"a hotdog in a tutu skirt"
"a llama wearing a suit"
"a model of the eiffel tower made out of toothpicks"
"a mug of hot chocolate with whipped cream and marshmallows"
"a pair of tan cowboy boots"
"a palm tree, low poly 3d model"
"a panda wearing a necktie and sitting in an office chair"
"a pigeon reading a book"
"a plate of fried chicken and waffles with maple syrup on them"
"a plate piled high with chocolate chip cookies"
"a plush t-rex dinosaur toy"
"a recliner chair"
"a red cardinal bird singing"
"a red rotary telephone"
"a roast turkey on a platter"
"a roulette wheel"
"a sliced loaf of fresh bread"
"a squirrel dressed like Henry VIII king of England"
"a squirrel dressed up like a Victorian woman"
"a stack of pancakes covered in maple syrup"
"a steaming basket full of dumplings"
"a steaming hot plate piled high with spaghetti and meatballs"
"a straw basket with a cobra coming out of it"
"a tiger dressed as a doctor"
"a tiger karate master"
"a tiger wearing sunglasses and a leather jacket, riding a motorcycle"
"a toilet made out of gold"
"a train engine made out of clay"
"a tray of Sushi"
"a typewriter"
"a very cool and trendy pair of sneakers"
"banana"
"Coffee cup with many holes"
"miniature schnauzer wooden sculpture"
"the Imperial State Crown of England"
"Wedding dress made of tentacles"


Acknowledgements: The website template was borrowed from Lior Yariv.