Cinematic AI Video With Stable Diffusion and LeiaPix

Key Takeaways:

How to create a cinematic Navi character
How to create dynamic backgrounds using LeiaPix
Seamlessly blending a character and background using a free Lightwrap tool

In this tutorial, I’ll show you how to create a seamless AI video, blending a dynamic background with your own AI-generated character. Whether you’re a beginner or have experience with AI video creation, this guide will help you achieve professional results.

5 Aspects That Affect the Consistency of AI Video

Before we dive into the tutorial, I want to share my experience on how five aspects of the base video can affect the consistency of AI video.

When I search for videos on platforms like Instagram or Pexels.com, I look for shots with a lot of depth of field. I noticed that in my Steampunk cat video, a blurry background with a clear character made the AI video look much better.

Video details are another crucial aspect. In one of my videos, the floor had lots of details, causing the AI video to flicker more. However, in a video with an even floor, there was less flickering, and the video remained consistent.

While working on my Lego person video, I realized it’s essential to pick videos where the character’s arms or hands don’t cover their face. Excessive movement makes the AI video less consistent.

Using a black background or green screen also improves video consistency. You can use tools like Magic Mask in Resolve or the green screen feature in RunwayML to remove the background. When the AI has fewer details to process, it creates a more consistent video.

Lastly, simplifying your outfit and ensuring your face movements are clear to the AI help improve consistency, especially when transforming into a character that looks very different from your own face.

Step 1: Create a Dynamic Background With LeiaPix

To create the dynamic background, I used LeiaPix. Here’s how:

Create a free account at LeiaPix.
Click on the upload button and select a still image.
Adjust the animation settings. I changed the animation length to 5 seconds, set the style to horizontal, and adjusted the motion to regular.
To download the image, click on the share icon, select the MP4 option, and save the file.

Step 2: Create a Navi Character Using Deforum in Stable Diffusion

Now, let’s create the Navi character using my free Deforum settings file.

Download my free settings file by following the link in the description.
Load the settings into the Deforum tab in Stable Diffusion.
If you haven’t installed Stable Diffusion or Deforum yet, or if you’re unfamiliar with installing models or Loras, check out the video links provided below for detailed instructions.

In the Run Tab, I set the sampler to Euler A, the width to 576, the height to 1024, and the steps to 20. In the Keyframes Tab, I adjusted the CFG scale to 3.

After that, I added my prompt in the Prompts Tab and pasted my negative prompt as well.

Next, I moved the information to the ControlNet Settings. Here’s the setup:

Enable the first ControlNet model.
Select Pixel Perfect, set the preprocessor to None, and choose the Control Tile model.
Set the weight schedule to 1 and the starting control step to 0.4.
For the second ControlNet model, don’t select Pixel Perfect, set the preprocessor to None, and choose Temporal Net. Leave the weight schedule to 1 and set ControlNet as more important.

Step 3: Use Free Lightwrap Tool to Seamlessly Blend Foreground With Background

Now, let’s blend the character with the background using my free Lightwrap tool. Here’s how:

Download the Lightwrap tool from the link below.
Open the Fusion Tab in DaVinci Resolve, then open the Effects Tab.
Navigate to Macros under the Tools section, and drag the settings file into the folder that appears.
In the Edit Tab, select the foreground and background images, right-click, and choose New Fusion Clip.
In Fusion, use the Magic Mask Tool to clear the background for the Navi character.

For users with the free version of DaVinci Resolve, you can also use the green screen feature in RunwayML or the depth map extension in Stable Diffusion to remove the background.

To use the Lightwrap tool, delete the merge node and drag the background and foreground into the appropriate inputs. You can adjust the blur and background overlap settings for a more subtle effect.

Step 4: Color Grading the Final Video

To match the foreground character with the background, use the Color Tab in Resolve:

Have both your Fusion Clip and the background video clip on the timeline.
Select the clip you want to grade, right-click on it, and choose Shot Match to This Clip.
The foreground will now be color-matched to the background.

Conclusion

By following these steps, you can create high-quality, consistent AI videos using Stable Diffusion and LeiaPix. I hope you found this tutorial helpful, and if you want to learn more about AI video creation, be sure to check out the other tutorials on my YouTube channel.