Innovate Futures @ Benji
patreon
Stable Diffusion ComfyUI Workflow - Using Multimodal Pipeline To Create AI Video
🕑 Added 2024-07-02 13:48:03 +0000 UTC
Video : https://youtu.be/EgM4OGRj2OQ
If you need to setup Local Ollama to host your local LLM with ComfyUI.
Here's previous tutorials :
1 - https://youtu.be/EQZWyn9eCFE
2 - https://youtu.be/yR2Y9G71w6E
We'll be exploring how to create stunning AI videos with the help of a multimodal pipeline.
Conceptual Diagram (Current Version)
As we all know, AI video models have been evolving rapidly, and now we have companies like Kling AI, Luma AI, and the latest Gen-3 AI video models from RunwayML. Although Runway Gen-3 is currently available only for text-to-videos, image-to-video capabilities are just around the corner.
In this workflow, I'll show you how to transform natural language content into Stable Diffusion Text Prompts, which can be used to generate images for each scene. These images serve as the initial keyframes for the AI video generator. We'll be using powerful models like LLaMA 3 fine-tuned SD Prompt large language models to transform the storyline into text prompts for image generation in Stable Diffusion.
But that's not all! We'll also explore the integration of ControlNet, a tool that allows you to generate specific actions and movements in your scenes. Whether you're using Luma AI, Kling AI, or RunwayML, once image-to-video capabilities are enabled, you can use these starting keyframes to generate captivating video scenes.
Throughout the video, I'll walk you through the different settings and custom nodes in ComfyUI, including the face detailer, upscalers, and reactor face swap. We'll also discuss the importance of scene descriptions and character styles in creating high-quality AI videos.
Goodies For Patreon Supporters: https://www.patreon.com/posts/create-story-llm-107317948
Comments
Benjamin Law
For Reactor nodes, you have to follow their installation guide for ComfyUI. Some custom nodes required additional steps in setup, it cannot be 1 click install via ComfyManager. Heres the guide : https://github.com/Gourieff/comfyui-reactor-node?tab=readme-ov-file#installation
ChimpinJoy
Heyy, the workflow includes Reactor Node which cannot be installed through Comfy Manager. It checks for updates only. How should we proceed? Thank you
