Innovate Futures @ Benji
patreon
Stable Diffusion ComfyUI Lightweight Workflow For Animated Video
🕑 Added 2024-01-29 13:20:23 +0000 UTC
VIDEO:https://youtu.be/Wtjcr5EtMCg
We have developed a lightweight version of the Stable Diffusion ComfyUI workflow that achieves 70% of the performance of AnimateDiff with RAVE. This means that even if you have a lower-end computer, you can still enjoy creating stunning animations for platforms like YouTube Shorts, TikTok, or media advertisements.
In this video, we take you through the new workflow step by step. We start by loading our checkpoints and setting the clip layers. We then proceed to the IP adapter, which is a basic and fast-loading component designed for lightweight usage. Next, we explore the control net, which focuses on minimalistic control of character movements using the DW pose model. After that, we load the source videos and apply resizing and preprocessing techniques. Finally, we delve into the sampling steps, using the LCM method and applying latent upscales to enhance details and colors.
By following this simple and quick workflow, you can generate animations with different styles without being influenced by the characters or backgrounds of the source videos. The key is to prepare a high-quality, bright, and detailed image in the IP adapter for styling. This image will serve as a reference for the animations. We demonstrate various examples in the video, showcasing the results achieved with different IP adapter images. Although this workflow may not match the quality of the complete element diff workflow, it provides a lightweight and fast solution that is accessible to users with lower computing power.
Things You Need For This Workflow Default Setting:
Checpoints Model: Any checkpoint you prefer
Lora animatediff/v3_sd15_adapter.ckpt
https://huggingface.co/guoyww/animatediff/blob/main/v3_sd15_adapter.ckpt
AnimateDiff Model: CiaraRowles/TemporalDiff
https://huggingface.co/CiaraRowles/TemporalDiff/blob/main/temporaldiff-v1-animatediff.ckpt
ControlNet: DWPose Preprocesser With OpenPose :
https://huggingface.co/lllyasviel/sd-controlnet-openpose
Workflow : In This Post Attachment
Comments
Benjamin Law
I am using VHS load video and combine video. We did not use AnimateDiff Combine video load. That one is discontinuing.
Benjamin Law
You have to download and install the VS Studio again .
Nelson
Hi Benji, when I queue prompt, the comfy ui got disconnect and here is the error message from my terminal: To see the GUI go to: http://0.0.0.0:8188 FETCH DATA from: /home/nelson/ComfyUI/custom_nodes/ComfyUI-Manager/extension-node-map.json [AnimateDiffEvo] - WARNING - This warning can be ignored, you should not be using the deprecated AnimateDiff Combine node anyway. If you are, use Video Combine from ComfyUI-VideoHelperSuite instead. ffmpeg could not be found. Outputs that require it have been disabled got prompt Killed Any idea how to solve the issue?
polaraju p
Error occurred when executing DWPreprocessor: Unknown C++ exception from OpenCV code this error came please tell me
Benjamin Law
Nice! A100 love it, I want to buy one also. hehe :) So, which one are you running for 20 mins? The RAVE+ AnimateDiff or the AnimateDiff Full version? Usually, it does takes time to run animate video generation. It's generating frame by frame, most of the time the system are consuming on that. The difference of this one compare with the other 2 workflow(RAVE+animatediff and Animatediff Full Ver.) , this workflow take less memory caching in the VRAM because of less models, controlnet, and sampling processers. If try to speed up with A100, I suggest tranc the frame size smaller. Then scale up later. But then that also consum your time manually to upscale it or another processing time by AI to upscale it.
bw
Hello Benji, first of all thank you for these great posts. They are all amazing. I'm just curious about how to speed up the workflow. I use A100 80GB and the previous workflow takes about 20 minutes. This workflow also takes 15 minutes. Is it normal that it takes this long? Do you have any suggestions on how I can optimize the A100 to shorten the workflow runtime? Thank you very much.
