sebastiankamph
patreon
LLM in Comfy. Automatic prompts!
🕑 Added 2024-08-27 09:39:24 +0000 UTCDOWNLOAD example workflow in bottom of post!
Video guide here: https://youtu.beMYSjvwD_s
With these nodes you will connect a LLM to Comfy and get great prompts for your images. Especially great for Flux, which is trained for natural language.
How to install
Go into your manager
Select "Custom Nodes Manager"
Search for Searge LLM and press Install
Download https://huggingface.co/MaziyarPanahi/Mistral-7B-Instruct-v0.3-GGUF/resolve/main/Mistral-7B-Instruct-v0.3.Q4_K_M.gguf
Create the directory and place model in /ComfyUI/models/llm_gguf/
Restart your Comfy
DONE!
These are the nodes available and how they connect. 
From the generate string, you connect to your CLIP Text encoder (Prompt). To be able to connect to the CLIP Text Encode, you can right click on that node and select "Convert widget to input"
As per the instructions from Searge's Github:
Configure the Searge_LLM_Node with the necessary parameters within your ComfyUI project to utilize its capabilities fully:
text: The input text for the language model to process.
model: The directory name of the model within models/llm_gguf you wish to use.
max_tokens: Maximum number of tokens for the generated text, adjustable according to your needs.
instructions: The instructions for the language model to generate a prompt. It supports the placeholder {prompt} to insert the prompt from the text input. Example: Generate a prompt from "{prompt}"
Advanced Options Node
The Searge_AdvOptionsNode offers a range of configurable parameters allowing for precise control over the text generation process and model behavior.
The default values on this node are also the defaults that Searge_LLM_Node uses when no Searge_AdvOptionsNode is connected to it.
Below is a detailed overview of these parameters:
Temperature (temperature): Controls the randomness in the text generation process. Lower values make the model more confident in its predictions, leading to less variability in output. Higher values increase diversity but can also introduce more randomness. Default: 1.0.
Top-p (top_p): Also known as nucleus sampling, this parameter controls the cumulative probability distribution cutoff. The model will only consider the top p% of tokens with the highest probabilities for sampling. Reducing this value helps in controlling the generation quality by avoiding low-probability tokens. Default: 0.9.
Top-k (top_k): Limits the number of highest probability tokens considered for each step of the generation. A value of 0 means no limit. This parameter can prevent the model from focusing too narrowly on the top choices, promoting diversity in the generated text. Default: 50.
Repetition Penalty (repetition_penalty): Adjusts the likelihood of tokens that have already appeared in the output, discouraging repetition. Values greater than 1 penalize tokens that have been used, making them less likely to appear again. Default: 1.2.
These parameters provide granular control over the text generation capabilities of the Searge_LLM_Node, allowing users to fine-tune the behavior of the underlying models to best fit their application requirements.
ALTERNATE Manual install from GIT Url
Paste & install https://github.com/SeargeDP/ComfyUI_Searge_LLM.git
If this gives you an error about permissions, you need to change your security settings.
Go into \ComfyUI\custom_nodes\ComfyUI-Manager\
Open config.ini
Change security_level = weak
Save the file
Comments
Potap Biolap
I haven't gotten around to installing the nodes yet. Probably the conflict is because my portable ComfyUI is running Python 3.12 and this node needs 3.10 to work, but I can't check as I don't know how to change to 3.10 in the portable version. Thanks anyway.
Sebastian Kamph
Import failed means your custom nodes fail to load. That can be because they or comfy is either out of date, or they are conflicting with another node. Make sure to update everything. If they still fail, they are conflicting with other installed nodes and you must start removing nodes to see which conflict.
Potap Biolap
I apologize, I am new to using ComfyUI. I am trying different things, including what is described here. But I have a problem if I do as described in the instructions I get (IMPORT FAILED). Perhaps this has already been discussed somewhere? I found only here: https://github.com/SeargeDP/ComfyUI_Searge_LLM/issues/11 but I still don't understand what I need to do.
Thomas Ruhmann
Hi Sebastian, I have installed the workflow in ComfyUI env in ThinkDiffusion. The workflow creates the images but I don´t get the output of the generated prompt in the Searge Output Node, neither in the added for the original prompt. Any ideas? It looks like I dont have a text box like in the video under text
Good Dude
Got it, now just messed it all up trying to ad animate anyone, ugh. lol
Sebastian Kamph
Troubleshooting further down in this post: https://www.patreon.com/posts/flux-how-to-for-109332332
Good Dude
cannot seem to get the checkloader NF4
steamrick
Whelp, that's another 4GB VRAM gone unless it loads and unloads the LLM every single time...
