sebastiankamph
patreon
Multiple ControlNet workflow for ComfyUI
🕑 Added 2024-06-11 21:47:18 +0000 UTCIn this guide we’ll go over a workflow for using ControlNet within Comfy. You will be able to easily use one or more ControlNets just by flipping a switch. ControlNet will control your output image by the input image you load.
If you don't have Comfy or the Comfy Manager installed, you can find a guide on it here: https://www.youtube.com/watch?v=KTPLOqAMR0s
Drag and drop the .json file (download at bottom) into your Comfy. If you get red nodes, make sure to go into your Manager and Install all missing custom nodes.
Install ControlNet models by clicking Install Models and then search for ControlNet. Make sure they are of type ControlNet and base 1.5 (they will be named control_v11p_sd15_lineart.safetensors or similar)
First, load your preferred image in the Load image node and select the size you want your output image to be.
Select your preferred model. By default in this workflow, an 1.5 model is recommended. Here, we’re using epicrealism, it can be downloaded here https://civitai.com/models/25694?modelVersionId=160989
Load a default VAE like 840000. If you don’t have a VAE, go into your Manager -> Install models -> Search for 840000 and press install
If you want to load a Lora, you can select these from the Load Lora nodes. To disable or enable loras, right click a node and select “Bypass node” or select it and press Ctrl + B / Cmd + B. When they are purple, they are disabled.
Type your prompt in the positive (green) and negative (red) nodes. In the example we use “Man closeup portrait” as our positive prompt ie. what we want to see in our image.
These nodes are your ControlNets and they are split up into several groups. First, a ControlNet model is loaded. We’re also loading a preprocessor to look at the input image and send that image preprocessed together with the ControlNet model into the Apply ControlNet node. This is where you would finetune your settings if needed.
Strength: How much impact this ControlNet will have on your result, set 0-1
Start_percent: When in the generation ControlNet will activate, set 0-1
End_percent: When in the generation ControlNet will finish, set 0-1
Example A: If your generation is running for 20 steps (Ksampler value) and you set end_percent to 0.5, then ControlNet will be active for 50% of that generation time, 10 steps. The last 10 steps will run without ControlNet.
Example B: Your generation is set at 20 steps in the Ksampler. Start_percent is set at 0.5 and end_percent at 0.8. Your generation will run 10 steps without ControlNet. It will activate after 10 steps and run with ControlNet and then disable again after 16 steps to finish the last 4 steps without ControlNet.
To enable or disable a ControlNet group, click the “Fast Bypasser” node in the right corner which says Enable yes/no
Here’s an example of a disabled ControlNet through the bypasser.
Your output will be in this group together with the Ksampler. The Ksampler is where the actual generation or rendering takes place.
Set the amount of steps you want. Around 10-15 for euler_a or 15-25 for dpm2m-karras
CFG is based on the model you use. Most models use between 3-7, but specialty models like LCM/Lightning/Turbo etc. will use a lower CFG like 1-2. Check your model information.
Denoise is how much of your input image will change. This is not based on ControlNet, but more similar to img2img. If you lower the denoise value, you will retain parts of your latent_image input. However in this workflow our latent_image input is just an empty node so nothing will be retained. Keep value at 1.
Seed is set at randomize, and gives you a random image each time you click generate. If you want to finetune your settings, you can set this to fixed to see the changes you make for each generation.
As an added bonus, there is an ultimate SD Upscale node to upscale your results. Recommended is to leave this disabled at first, as the extra generation time will be quite resource intensive.
Feel free to play with the values to adjust the upscaling, try different upscaling models, steps and samplers to find the settings for your image.
It currently is upscaling in tiles. Recommended value for tiles are 512x512 or 768x768.
Comments
Sebastian Kamph
Glad to hear it works well for you :)
Chariti Canny
I also was doing a conditioning concat on the 2 controlnets, which "worked" but also, not great results. I'm not sure WHAT it was doing (technically speaking) in that configuration.
Chariti Canny
This is really amazing. I was trying all day to do multi control nets, but never had a controlnet_aux connected. I got interesting results but always overbaked/over-detailed. And I was using sdxl. Once I found your vid and switched to 1.5, added the aux, the results are better than I could imagine. TYSM for making this video and workflow.
Jasmine Ayling
my bad, its all good
