Innovate Futures @ Benji
patreon
ComfyUI Workflow SEG And Unsampling AnimateDiff - For Patreon Supporters
🕑 Added 2024-09-27 13:00:17 +0000 UTC
Tutorial Video : https://youtu.be/J7nuNjDwVx8
Previous Progress Of This Workflow : https://www.patreon.com/posts/coming-up-i-went-1127
This is another AnimateDiff workflow in ComfyUI.
The purpose of this workflow is to understand how to use unsampling steps for v2v animation, and style transfer of the selected Segment mask region.
Using Unsampling method in the first sampling , we are able to create more consistency objects in AnimateDiff. Similar effect as RAVE as what I talked about before, but Unsampling consume less GPU power.
Visual object tracking with Point Editor
We have a very good tool, Point Editor for SAM2. So without text prompting in Florence 2 or so , and hoping the BBOX can be highlight the object we want to track.
Now, we can have a more effective and intuitive way, by visually pointing the object we want to segment.
SamplerCustom , Unsampling and Add IPA for Sampling
We are basically leveraging the IPAdapter and Mask for the style transfer as usual in AnimateDiff workflow we have walkthrough before.
Second Sampling - Refine edges on the SEG object
Like I mentioned in the video (https://youtu.be/J7nuNjDwVx8), the Umsampling method create consistency based on source video, but it's appearing diffusion noise in some part of the Segment object. (appear some flicker blue dots noise)
Therefore, we use CannyEdge and Blur Mask outline to refine the animation and solve this problem.
Follow the Tutorial Video how we configurate before run the generation process. You will have no problem. But this workflow is kind of advance, you have to understand what is diffusion noise, and how those connections on the workflow works before you can get some good result.
Workflow file attached in this post below, if you have question , feel free to post in the comment section.
We all keep learning everyday.
And Have Fun!
Comments
Benjamin Law
Please check the video tutorial and the content in this post. Let me know in specific detail, "what and how it is not generate as provided image". So we can pinpoint on the way. Thanks
Hein Latt
video is not generate as provided image on foreground seg, can you guide me?
Benjamin Law
models and nodes are in the workflow setting. you can check it out.
Cezary Korpys
what are the sources of models and nodes
Clayfacer
getting good. exactly why I subscribe
