Innovate Futures @ Benji
patreon
ComfyUI With Meta Segment Anything Model 2 - Tutorial
🕑 Added 2024-08-02 14:00:15 +0000 UTC
Tutorial : https://youtu.be/O8QdvZbRDp4
For Patreon Supporters Additional Contents : https://www.patreon.com/posts/109322698/?utm_source=youtube&utm_medium=video&utm_campaign=20240802
In this exciting video, we delve into the cutting-edge realm of artificial intelligence and computer vision with Meta's Segment Anything Model 2, also known as SAM 2. This next-generation AI model revolutionizes object segmentation, offering real-time capabilities for both images and videos. SAM 2's advanced architecture, featuring a unique memory mechanism, enables precise segmentation even in challenging scenarios like occlusions and reappearances. With its state-of-the-art performance, SAM 2 is a game-changer in the field of AI-driven object segmentation.
Previous Segement Anything 1 Multi Objects Editing Example :
https://www.youtube.com/shorts/NNcihC0ZRGk
Discover the power of SAM 2 as we guide you through implementing this groundbreaking technology in ComfyUI. From adding stunning effects to videos to tracking objects with ease, SAM 2 proves to be a versatile tool for content creators, researchers, and engineers alike. With SAM 2's open-source nature and accessibility under the Apache 2.0 license, it paves the way for innovation and experimentation in the AI community, from hobbyists to seasoned researchers. Join us as we explore the limitless possibilities of SAM 2 in enhancing image editing, video object tracking, and AI animations.
Unleash the potential of SAM 2 in your projects by following our step-by-step tutorial on integrating this powerful AI model into ComfyUI. Witness how SAM 2 collaborates seamlessly with other custom nodes and large language models to accurately segment objects in videos and images. Whether you're a novice looking to explore the world of AI or a seasoned professional seeking advanced segmentation tools, SAM 2 offers a user-friendly and efficient solution. Join us on this journey of exploration and innovation with SAM 2, the future of object segmentation in artificial intelligence.
segment-anything-2
https://ai.meta.com/blog/segment-anything-2/
https://github.com/kijai/ComfyUI-segment-anything-2
https://github.com/facebookresearch/segment-anything-2
Model : https://huggingface.co/Kijai/sam2-safetensors/tree/main
Save to ComfyUI/models/sam2
Attached workflow of this tutorial :
Comments
Benjamin Law
Honestly, not being rude or something. But consider your membership level before asking for help here. All terms are list on the page very clearly. thanks. If everyone come and ask ask ask, I don't have all the time for freebies stuff, as I have my own company to run.
Ritik Agrawal
Hi @Benji. I saw the tutorial and it looks great. Can I use this workflow to implement a virtual try on setup? Like using SAM2 + IP Adapter? Also while doing so, can I preserve the quality/details/texture of the cloth that I want to try on, I saw that it can lead to an animated look. Can you please help me on that?
Benjamin Law
as the error message , need install torch>=2.1.1. and it can be done by running the requirements.txt
Nico Angelone
I've updated (supposedly) pytorch via Comfy Manager feature "install pip packages", not sure about that tho... any clue?
Nico Angelone
Hi @Benji! great tut! the only thing is that I've run into an issue about Florence-2. So far I dowloaded the .bin model from HF and placed into comfyUI/models/microsoft/ but i got the following error: Error occurred when executing Florence2ModelLoader: PyTorch SDPA requirements in Transformers are not met. Please install torch>=2.1.1. File "/home/user/ComfyUI/execution.py", line 151, in recursive_execute output_data, output_ui = get_output_data(obj, input_data_all) File "/home/user/ComfyUI/execution.py", line 81, in get_output_data return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True) File "/home/user/ComfyUI/execution.py", line 74, in map_node_over_list results.append(getattr(obj, func)(**slice_dict(input_data_all, i))) File "/home/user/ComfyUI/custom_nodes/ComfyUI-Florence2/nodes.py", line 173, in loadmodel model = AutoModelForCausalLM.from_pretrained(model_path, attn_implementation=attention, device_map=device, torch_dtype=dtype,trust_remote_code=True) File "/home/user/miniconda3/envs/comfyui/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 558, in from_pretrained return model_class.from_pretrained( File "/home/user/miniconda3/envs/comfyui/lib/python3.10/site-packages/transformers/modeling_utils.py", line 3620, in from_pretrained config = cls._autoset_attn_implementation( File "/home/user/miniconda3/envs/comfyui/lib/python3.10/site-packages/transformers/modeling_utils.py", line 1478, in _autoset_attn_implementation config = cls._check_and_enable_sdpa( File "/home/user/miniconda3/envs/comfyui/lib/python3.10/site-packages/transformers/modeling_utils.py", line 1650, in _check_and_enable_sdpa raise ImportError(
