Innovate Futures @ Benji
patreon
AnimateDiff ControlNet LCM Flicker-Free Animation Video Workflow
🕑 Added 2023-12-06 16:33:40 +0000 UTCHey there! Let me break down the features and functions of this workflow for you.
Any questions about the setting, feel free to ask in this post comment.
The ComfyUI workflow is designed to make the AnimateDiff process smoother and more efficient. It offers a user-friendly interface that allows you to easily load your source videos and connect various custom nodes. With the checkpoint loader, you can load your desired checkpoint model, such as the Realistic Visions 5.1 with VAE checkpoint model. This helps ensure accurate and high-quality results.
One of the key functions of the ComfyUI workflow is the Lora loader. This custom node enables you to load the LCM Lora model, which is essential for generating the animation. Depending on your checkpoint model version, you can choose the appropriate LCM Lora model, such as the LCM SDXL Lora or LCM SD1.5 Lora. This flexibility ensures compatibility and optimal performance.
Another notable feature is the ability to incorporate upscale groups and the IP adapter. By using upscale groups, you can enhance the image quality of your animations. This involves connecting the VAE decode output to an upscaler model, such as the 4x UltraSharp, to improve the video's resolution, color, and sharpness.
Additionally, the integration of the IP adapter allows for further customization and improvement. By adjusting the sampling settings, such as the sampling step and denoising values, you can fine-tune the animation and achieve even better results. This feature gives you more control over the output and allows for experimentation and creativity.
Overall, the ComfyUI workflow combines the power of different custom nodes, such as the checkpoint loader, Lora loader, upscale groups, and the IP adapter, to provide a comprehensive and efficient solution for generating high-quality animations with the Animate Diff technique. It's user-friendly, flexible, and offers various customization options to help you create captivating and visually stunning videos.
Comments
bo setty
hi Benji Can comfyui do this https://youtube.com/shorts/EoEn_bf_ZI0?feature=share
Benjamin Law
in ContorlNet , set the Strength from 0.1 to 1
Benjamin Law
about 10 to 15 minutes, it's really depends on how many frames you are doing. But usually, for my 20 to 30 seconds animation, that is my case. And yes, don't touch any thing the openpose part goes green, and let it run.
Zenvendof Zeno
Hi Benji, any idea with a 4090, how long does controlnet openpose takes, it seems like it gets struck there for a long time (OpenPose Pose Recongition) is highlighted in green, no error messages and the GPU isnt very high. How to enable debug or look at any error messages? ControlNet is working as the Lineart portion works and generated the temp files. Thanks alot and appreciate the tutorials!
Ninja Chef
how to set that prompt is more important than controlnet
bo setty
Hello, I would like to ask how to use comfyui to generate beauty dancing video, how to make the background not follow the prompt word change or how to use the adapter to change the background of the video
Benjamin Law
For NUM of frames you can try 24fps it looks smoother than 12. And yes the calculation is correct.
bo setty
Hello, I have solved this problem and reinstalled comfyui to solve this problem. I would like to ask the following questions. For example, 1.if I have a 20-second video, should I write the frame of comfyui video 20*12=240 frames or how to calculate it? 2.Take a look at the dancing video I provided to convert the middle of the hair is a little deformed how to repair this,? linkURL:https://youtube.com/shorts/gpqKmxScjjM?feature=share 3, video frame dynamic change background and change the color of clothes in the prompt words should be reflected or how to write it can give a demo
Benjamin Law
in the beginning, Input file range : out of range. Have you miss some frame input? And from my experience, last time I got this message , caused VideoCombine, AnimateDiff are not update. There are some new value need to be set, if you have the new version of all these Custom Nodes, you will see it.
bo setty
Here's my link to run comfyui https://youtu.be/m3OZzu_UUwk https://youtu.be/m3NrT2IwGgc And I execute our workflow link error you look at what happened I repeated 3 times for several hours, the same exception error interrupts the operation help me check thank you
Benjamin Law
Hi, have you update your ConfyUI and all Custom Nodes? The error message looks like you are missing a value in it.
bo setty
Is it out of memory? What should I do
bo setty
Hello, I bought this member of yours and when I ran your workflow, I had this error widget_values = self._widget_values_to_dict(v.type, v.widgets_values) 1.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! File "I:\comfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI_Ib_CustomNodes\script\__init__.py", line 101, in _widget_values_to_dict return {name: value for name, value in zip(widget_value_names, widget_values)} TypeError: 'types.SimpleNamespace' object is not iterable Traceback (most recent call last): File "I:\comfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI_Ib_CustomNodes\__init__.py", line 39, in pnginfo_add_text_hook comfy_script = script.WorkflowToScriptTranspiler(workflow).to_script(end_nodes) File "I:\comfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI_Ib_CustomNodes\script\__init__.py", line 283, in to_script c += self._node_to_assign_st(self.G.nodes[node]) 2.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! and could not patch. key doesn't exist in model: diffusion_model.middle_block.2.in_layers.2.weight could not patch. key doesn't exist in model: diffusion_model.middle_block.2.emb_layers.1.weight could not patch. key doesn't exist in model: diffusion_model.middle_block.2.out_layers.3.weight could not patch. key doesn't exist in model: diffusion_model.output_blocks.2.1.conv.weight could not patch. key doesn't exist in model: diffusion_model.output_blocks.5.2.conv.weight could not patch. key doesn't exist in model: diffusion_model.output_blocks.8.2.conv.weight ' 3.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Finally, when we run to the last node to change the face（reactor fast face swap）, an exception occurs and the process ends without generating anything !!!!!!!!! Exception during processing !!! Traceback (most recent call last): File "I:\comfyUI\ComfyUI_windows_portable\ComfyUI\execution.py", line 153, in recursive_execute output_data, output_ui = get_output_data(obj, input_data_all) File "I:\comfyUI\ComfyUI_windows_portable\ComfyUI\execution.py", line 83, in get_output_data return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True) File "I:\comfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI_ezXY\autoCastPatch.py", line 299, in map_node_over_list return _map_node_over_list(obj, input_data_all, func, allow_interrupt) File "I:\comfyUI\ComfyUI_windows_portable\ComfyUI\execution.py", line 76, in map_node_over_list results.append(getattr(obj, func)(**slice_dict(input_data_all, i))) File "I:\comfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\comfyui-reactor-node-main\nodes.py", line 158, in execute out_images = np.ndarray(shape=image_np.shape) numpy.core._except How can I solve it
