Gen AI for Production

Pioneering the Next Generation of VFX Technology

Animation

3D Matte Painter & Environment Artist

I am a Matte Painter and Environment Artist for feature for live action and animated films, creating everything from concept development and environment creation to matte painting and 3D-projected environments. I specialize in innovating with new technologies and developing efficient workflows for environment creation.

Generative AI has opened exciting new possibilities for visual development and content creation, but most text-to-video and image-to-video tools remain too unpredictable for the demands of professional production. In feature film and animation, the real challenge is not just generating images or video—it is achieving control, consistency, and art direction throughout the process. A production-ready AI workflow must support deliberate visual development, not random output.

Local AI System Setup Used in Workflow

AI Models	Software
Flux.1, Flux.2 Nano Banana Llama QwenVL Trellis.2 Hunyuan3D ChatGPT WAN2.2 Skyreels	ComfyUI AI-Toolkit Prism Pipeline Procreate Blender Photoshop Docker NVIDIA AI Workbench

	Primary Workstation	Headless Render Node
OS	Windows 11	Rocky Linux 8.10
CPU	Ryzen 9 7900X 12 Core	Ryzen 9 7900X 12 Core
RAM	64 GIG	64 GIG
GPU	RTX 5090 32 GIG VRAM	RTX 5090 32 GIG VRAM

Designing the Input

In a production-focused generative AI workflow, art direction begins with carefully designed creative inputs. For this piece, I created a hand-drawn character sketch in Procreate on the iPad, using it as a visual foundation to guide the AI in generating a character that appears consistently across multiple shots.

Character Lock

Using Black Forest Labs’ FLUX.2 diffusion model, I translated the concept illustration into a photoreal character design. I then used Google’s Nano Banana image-to-image model to generate the character across turnarounds, action poses, varied lighting conditions, and close-up details. A dataset of 130 images was used to train the FLUX.2 LoRA in AI-Toolkit, helping maintain visual consistency and character fidelity throughout the process.

ControlNet Character Rig

Using Microsoft’s Trellis and Hunyuan3D, I quickly generated a 3D model from reference imagery captured from multiple views. The resulting mesh was then separated into articulated segments, and I built an IK rig for the character in Blender. This rig was used to produce the control renders needed to direct the generative AI process with greater precision and consistency.

ControlNet Env Assets

Using a Trellis image-to-mesh model alongside NVIDIA’s 3D Object Generation Blueprint, powered by Trellis and Llama 3.1 8B Instruct, I rapidly generated environment assets that were used to produce control images, helping direct the AI’s output in both style and composition.

ControlNet Layout

The layout is established by arranging the AI-generated geometry into a full environment, animating the character rig, and positioning the shot cameras within the scene. Built in Blender, this 3D setup becomes the foundation for all control images, ensuring consistency in composition, staging, and guidance for the generative AI workflow.

Keyframe Generation

Using the trained character LoRA, prompts, ControlNets, and inpainting techniques with FLUX.2 in ComfyUI, I generated an overscan master scene image that established the look of the sequence and drove visual consistency across the AI image generation process. Depth and Canny control renders from Blender were used to position the environment and character within the start and end keyframes, providing a strong compositional foundation for the generation. These keyframes were then used to guide the AI video generation.