Innovate Futures @ Benji
patreon
AnimateDiff SAM2 multiple objects IPA Workflow -
🕑 Added 2024-08-02 14:00:16 +0000 UTC
Tutorial : https://youtu.be/O8QdvZbRDp4
Attached Workflow is a lightweight AnimateDiff workflow, and demonstrate how to use SAM2 and Florence 2 for segmentation in an AI animation scenario.
Not gonna explain too much about AnimateDiff in this tutorial , because I did too many in previous videos and posts. Scroll back to old post if you missed it.
Related To AnimateDiff :
https://www.patreon.com/collection/393434?view=expanded
https://www.patreon.com/aifuturetech/posts?filters%5Btag%5D=AnimateDiff+Workflows
Segment-anything-2
https://ai.meta.com/blog/segment-anything-2/
https://github.com/facebookresearch/segment-anything-2
Custome Node https://github.com/kijai/ComfyUI-segment-anything-2
Model To Download : https://huggingface.co/Kijai/sam2-safetensors/tree/main
Save to ComfyUI/models/sam2
Comments
Brian G
Thank you!
Benjamin Law
try this one : https://github.com/neverbiasu/ComfyUI-SAM2 You can refer to my other previous AnimateDiff workflows and series of tutorial. I was using this Segment Anything node, and it preform better on objects. The one from KJ Florence 2 + SAM2 are doing for character, but the way he designed the node can't do well on object.
Brian G
Hi Benji! Thanks for providing this workflow. I have been diving into it, but I am having trouble getting the SAM2 model to mask the shirt in my footage correctly. It's a two tone light pink floral shirt with a heavy pink collar and button fabric. I enter the keywords, and Florence does the correct bounding box and identifies the entire shirt, but when it goes through SAM2 and creates the mask, it will just mask the collar and button fabric, ignoring the light pink floral pattern of the rest of the blouse. Any suggestions on what I might be doing wrong? Thanks in advance!
