Aron Petau - generativeZola2023-06-20T00:00:00+00:00https://aron.petau.net/tags/generative/atom.xmlStable Dreamfusion2023-06-20T00:00:00+00:002023-06-20T00:00:00+00:00
Aron Petau
https://aron.petau.net/project/stable-dreamfusion/<h2 id="Stable_Dreamfusion">Stable Dreamfusion</h2>
<div class="sketchfab-embed-wrapper"> <iframe title="Stable-Dreamfusion Pig" frameborder="0" allowfullscreen mozallowfullscreen="true" webkitallowfullscreen="true" allow="autoplay; fullscreen; xr-spatial-tracking" xr-spatial-tracking execution-while-out-of-viewport execution-while-not-rendered web-share width="800" height="600" src="https://sketchfab.com/models/0af6d95988e44c73a693c45e1db44cad/embed?ui_theme=dark&dnt=1"> </iframe> </div>
<h2 id="Sources">Sources</h2>
<p>I forked a popular implementation that reverse-engineered the Google Dreamfusion algorithm. This algorithm is closed-source and not publicly available.
You can find my forked implementation <a rel="external" href="https://github.com/arontaupe/stable-dreamfusion">on my GitHub repository</a>.
This version runs on Stable Diffusion as its base process, which means we can expect results that might not match Google's quality.
The <a rel="external" href="https://dreamfusion3d.github.io">original DreamFusion paper and implementation</a> provides more details about the technique.</p>
<iframe
class="youtube-embed"
src="https://www.youtube-nocookie.com/embed/shW_Jh728yg"
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
referrerpolicy="strict-origin-when-cross-origin" allowfullscreen>
</iframe>
<h2 id="Gradio">Gradio</h2>
<p>I forked the code to implement my own Gradio interface for the algorithm. Gradio is a great tool for quickly building interfaces for machine learning models. No coding is required for the end user - they can simply state their wish, and the system will generate a ready-to-be-rigged 3D model (OBJ file).</p>
<h2 id="Mixamo">Mixamo</h2>
<p>I used Mixamo to rig the model. It's a powerful tool for rigging and animating models, but its main strength is simplicity. As long as you have a model with a reasonable humanoid shape in a T-pose, you can rig it in seconds. That's exactly what I did here.</p>
<h2 id="Unity">Unity</h2>
<p>I used Unity to render the model for the Magic Leap 1 headset.
This allowed me to create an interactive and immersive environment with the generated models.</p>
<p>The vision was to build an AI Chamber of Wishes:
You put on the AR glasses, state your desires, and the algorithm presents you with an almost-real object in augmented reality.</p>
<p>Due to not having access to Google's proprietary source code and the limitations of our studio computers (which, while powerful, aren't quite optimized for machine learning), the results weren't as refined as I had hoped.
Nevertheless, the results are fascinating, and I'm satisfied with the outcome.
A single object generation in the environment takes approximately 20 minutes.
The algorithm can be quite temperamental - it often struggles to generate coherent objects, but when it succeeds, the results are quite impressive.</p>