Director, VFX artist, and CEO Masaki Mizuno at Tokyo prodco Khaki, just dropped us his latest visual experiment, an hypnotic short film where the VFX result from an innovative pipeline where CG drives modified AI tools.

Masaki Mizuno: “I have always been drawn to noise as a form of visual expression. The scratched textures of analog film, the warped look of burned and damaged footage. Even after everything shifted to digital, I remained captivated by the strong presence of glitches and digital noise in RGB.

“So today, what is ‘noise’ in the context of AI? I believe one example is hallucination. With this film, I deliberately induced hallucination into live-action footage we shot of the dancer Toyotaka.

“For the city distortions, I created vertex animations in Blender and imported them into ComfyUI. To make this possible, I developed a custom Blender add-on that converts vertex animation data into JSON files readable by ComfyUI.
 

“I have always been drawn to noise as a form of visual expression… Even after everything shifted to digital, I remained captivated by the strong presence of glitches and digital noise.”

 
“The model used for the city deformation was Wan 2.1 ATI, which allows still images to move along defined coordinates. But I wanted to apply distortions to moving shots. So I 3D-tracked the live-action footage, reconstructed the camera’s movement in Blender by placing multiple reference objects, and added animated distortion objects at specific points in 3D space.

“Of course, when the background distorts, every person in the foreground must be rotoscoped. Unfortunately, AI-based roto tools weren’t accurate enough, so I had to mask everything manually — ironically, the part I most wish AI could automate.

“The ‘multiplying dancers’ sequence started from a hypothesis: Could AI face-swapping be used to replace the entire body? To test this, Toyotaka performed a variety of dance moves and expressions, which I used to train a LoRA model, allowing me to maintain consistent clothing, facial expressions, and motion characteristics.

“For motion capture, I used MOVE AI, which works with a single camera, and converted the dance into FBX data. In Blender, I applied 3D tracking, positioned motion-driven characters, and rendered them, then imported the output into ComfyUI to replace the CG characters with Toyotaka’s AI-generated body.

“I also incorporated AI relighting into the first shot using an open-source tool by NVIDIA called cosmos-transfer1-diffusion-renderer. I customized it but image degradation was still significant, so I could only use it selectively. That said, experimenting with it made me realize that in the near future, any video could be fully re-lit with AI — a glimpse of what’s to come.”

Watch the making-of video:
 
 

 
 

 
Production: Khaki

Director: Masaki Mizuno
AI/VFX: Masaki Mizuno
Director of photography: Shosuke Sasaki
Photography: Natsuaki Yoshida
Roto: Taupe
Tools development: Masahiro Teraoka
Producer/CEO : Shuntaro Okamoto (Vook)
Producer/Sales manager: Tomoko Kato (Vook)

Music: Kazuki Sugawara

Starring: Toyotaka
Body double: GeN