SD HeartMuLa

Multilingual AI music generation nodes for ComfyUI powered by the HeartMuLa model family. Generate full songs with lyrics in English, Chinese, Japanese, Korean, Spanish, and Russian.

Features

• Multilingual Lyrics - Generate music with vocals in English, Chinese, Japanese, Korean, Spanish, and Russian
• Song Structure Control - Use section markers like [Verse], [Chorus], [Bridge] to define song structure
• Style Tags - Control genre, vocal type, mood, tempo, and instruments
• 4-Minute Songs - Generate up to 240 seconds of continuous audio
• Zero-Shot Generation - No fine-tuning required, just provide lyrics and tags
• Modular Pipeline - Separate nodes for conditioning, sampling, and decoding

Nodes

| Node | Description | |------|-------------| | Model Loader | Downloads and caches HeartMuLa models | | Conditioning | Tokenize lyrics and tags into model conditioning | | Sampler | Generate audio tokens with CFG and temperature control | | Decode | Convert audio tokens to waveform using HeartCodec | | Tags Builder | Build style tags with genre, vocal, mood selection | | Transcribe | Extract lyrics from existing audio |

Installation

ComfyUI Manager

Search for "SD HeartMuLa" and install.

Manual

cd ComfyUI/custom_nodes
git clone https://github.com/singldev/comfyui_sd-heartmula.git
cd comfyui_sd-heartmula
pip install -r requirements.txt

Quick Start

1. Add SD HeartMuLa Model Loader and select 3B model
2. Connect to SD HeartMuLa Conditioning node
3. Enter lyrics with section markers:

   [Verse]
   Walking down the empty street
   Thinking about you and me
   
   [Chorus]
   We belong together
   Now and forever
   

4. Add style tags: pop, female vocal, energetic
5. Connect to Sampler → Decode → Preview Audio

Section Markers

HeartMuLa supports these official section markers:

| Marker | Usage | |--------|-------| | [Intro] | Opening instrumental or vocal intro | | [Verse] | Main verses | | [Prechorus] | Build-up before chorus | | [Chorus] | Main hook/chorus | | [Bridge] | Contrasting section | | [Outro] | Ending section | | [Instrumental] | Non-vocal sections |

Style Tags

Combine tags to control the output style:

• Genre: pop, rock, electronic, jazz, classical, hip-hop, r&b, country, folk, metal, indie
• Vocal: female vocal, male vocal, duet, choir, instrumental
• Mood: energetic, melancholic, uplifting, calm, aggressive, romantic, dreamy, dark
• Tempo: slow, medium, fast
• Instruments: piano, guitar, drums, synth, strings, etc.

Example: indie rock, male vocal, melancholic, slow, acoustic guitar

Models

| Model | Size | VRAM (fp16) | VRAM (4-bit) | Notes | |-------|------|-------------|--------------|-------| | 3B | ~6GB | ~12GB | ~6GB | Released, recommended | | 7B | ~14GB | ~24GB | ~12GB | Coming soon |

Models download automatically on first use to ComfyUI/models/HeartMuLa/.

File Structure

The node expects models to be placed in ComfyUI/models/HeartMuLa/. You can place custom fine-tunes or codecs here, and they will be detected by the Model Loader.

Standard Layout:

ComfyUI/
└── models/
    └── HeartMuLa/
        ├── HeartMuLa-oss-3B/       # Main Model (Transformer)
        │   ├── config.json
        │   ├── model.safetensors
        │   └── ...
        ├── HeartCodec-oss/         # Audio Codec
        │   ├── config.json
        │   ├── model.safetensors
        │   └── ...
        ├── tokenizer.json          # Shared tokenizer
        └── gen_config.json         # Generation config

Custom Models: To use a custom model or codec, simply place it in a subfolder.

• Select your model folder in the Model Version list.
• Select your codec folder in the Codec list.

ComfyUI/models/HeartMuLa/
├── My-FineTune-v1/         # Appears in "Model Version" list
├── My-Custom-Codec/        # Appears in "Codec" list (must contain "Codec" in folder name)

Memory Modes

The Model Loader includes a memory_mode option to optimize for different VRAM configurations:

| Mode | Description | Use Case | |------|-------------|----------| | auto | Auto-detect based on available VRAM | Recommended default | | normal | Full speed, no memory optimizations | 16GB+ VRAM | | low | Moderate memory savings | 10-16GB VRAM | | ultra | Aggressive memory cleanup | 8-10GB VRAM |

Requirements

• Python 3.10+
• 16GB RAM minimum (32GB+ recommended)

Supported Platforms

| Platform | Device | Notes | |----------|--------|-------| | NVIDIA GPU | CUDA | 12GB+ VRAM (or 6GB with 4-bit quantization) | | Apple Silicon | MPS | M1/M2/M3/M4 Macs supported | | CPU | CPU | Slow fallback option |

Platform Notes

• NVIDIA: CUDA 12.1+ recommended. 4-bit quantization available via bitsandbytes
• Apple Silicon: Uses Metal Performance Shaders (MPS). 4-bit quantization not available (CUDA only)
• CPU: Works but very slow, not recommended for generation

Parameters

Sampler Settings

| Parameter | Default | Range | Description | |-----------|---------|-------|-------------| | max_duration_sec | 60 | 10-240 | Maximum audio length | | temperature | 1.0 | 0.1-2.0 | Sampling randomness | | top_k | 50 | 1-500 | Top-k token filtering | | cfg_scale | 1.5 | 1.0-10.0 | Classifier-free guidance | | seed | -1 | -1 to 2^31 | Random seed (-1 = random) |

Credits

• HeartMuLa/HeartLib - Original model and codebase
• HeartMuLa/HeartCodec - Neural audio codec

License

Apache 2.0