Yet Another llama.cpp for ComfyUI

There are several bindings of ComfyUI to llama.cpp. In my opinion, all of them were too opinionated, as well as too out of date.

Philosophy

I've tried to make everything work fairly generically, as unopinionated as I can manage. There are no nodes for specialized tasks. For example, while this node set is capable of being used for prompt enhancement, there is no “prompt enhancement” node, because that's simply built from a chat.

All of the nodes include a generic kwargs parameter, which simply takes text. Rather than trying to anticipate every possible parameter, I've included only the most obvious parameters, and for anything else, use kwargs. kwargs can be empty (for no extra args) or an object in JSON format. Of course, this is only sufficient for those arguments that take JSON-compatible values. kwargs overrides other parameters.

As in HuggingFace, Pipelines are self-loading and include all components, but you can override the VAE and text encoder components if you wish, so they're optional inputs. If you want to load without these components, use kwargs to set them to null (None).

Workflows

(I'll probably add some soon, maybe)

Nodes

To perform a chat with llama.cpp, you need to load a model, create a conversation, run the completion, and then unpack the text from the resulting conversation. Because conversations are a wholly supported type, you can perform multi-stage chats, include system prompts, etc.

YALlama load model

Loads a GGUF model. Models should be in models/llamacpp. The seed set here is the default seed for future conversations.

Note that llama.cpp doesn't use PyTorch, so how it sees multiple GPUs may be different than how PyTorch does. You can select the GPU with main_gpu, and of course, you can split across multiple GPUs in any of llama.cpp's supported split modes.

YALlama create conversation

This node set treats llama.cpp principally as a chat completion model (rather than a simple text completion model), and so you need to create chats to complete. This node does so. In the usual case, you simply need one of these nodes to create the user's side of the conversation. But, you can chain multiple nodes together to create a longer conversation. For vision-language models, you can also include an image.

YALlama unpack conversation

Unpacks conversations, pulling out the text components. A chat completion creates a (more complete) chat, so this allows you to get the result of a chat completion.

YALlama chat completion

Performs an actual chat completion, inputting and outputting a conversation.