ComfyUI-QwenScope

ComfyUI nodes for Qwen-Scope — Qwen's released sparse-autoencoder (SAE) interpretability tooling for Qwen3 and Qwen3.5 LLMs. Capture residual streams, encode them through layer-specific SAEs, visualise feature activations, and steer generation, all inside a ComfyUI graph.

SAE interpretability is the practice of decomposing an LLM's residual stream into a sparse dictionary of human-interpretable "features." Qwen-Scope ships pre-trained TopK SAEs for every layer of seven Qwen base models, so you can read what the model is "thinking" at any layer — and nudge it.

Links

• Qwen-Scope collection: https://huggingface.co/collections/Qwen/qwen-scope
• Qwen-Scope explorer (live demo): https://huggingface.co/spaces/Qwen/QwenScope
• Technical report: https://qianwen-res.oss-accelerate.aliyuncs.com/qwen-scope/Qwen_Scope.pdf
• ComfyUI: https://github.com/comfyanonymous/ComfyUI

Supported models / SAEs

All 14 currently released Qwen-Scope packs:

| Family | Base model | Released SAE widths · L0 | | --- | --- | --- | | Qwen3.5 | Qwen/Qwen3.5-2B | W32K · L0_50 / L0_100 | | Qwen3.5 | Qwen/Qwen3.5-9B | W64K · L0_50 / L0_100 | | Qwen3.5 | Qwen/Qwen3.5-27B | W80K · L0_50 / L0_100 | | Qwen3.5 (MoE) | Qwen/Qwen3.5-35B-A3B | W32K · L0_50 · W128K · L0_100 | | Qwen3 | Qwen/Qwen3-1.7B-Base | W32K · L0_50 / L0_100 | | Qwen3 | Qwen/Qwen3-8B-Base | W64K · L0_50 / L0_100 | | Qwen3 (MoE) | Qwen/Qwen3-30B-A3B-Base | W32K · L0_50 · W128K · L0_100 |

L0 is the TopK sparsity (number of non-zero features kept per token); W is the SAE dictionary width. Higher L0 → denser, finer-grained features; higher W → larger feature vocabulary.

Pick any of these in the Load Qwen-Scope SAE node — the SAE pack is downloaded lazily on first use via huggingface_hub.

Install

Clone into your ComfyUI custom_nodes/ directory:

cd /path/to/ComfyUI/custom_nodes
git clone https://github.com/Nynxz/ComfyUI-QwenScope

Dependencies (transformers, huggingface_hub, safetensors, Pillow) are listed in pyproject.toml and install with pip install -e . if you want them pinned, otherwise ComfyUI's existing torch/PIL stack is enough for everything except transformers and huggingface_hub (which most ComfyUI installs already have for other nodes).

If you need a HuggingFace token (some repos may be gated), set HF_TOKEN in your environment before launching ComfyUI.

Nodes

All nodes live under the QwenScope category in the node menu.

| Category | Node | What it does | | --- | --- | --- | | Loaders | Load Qwen Model | Load a Qwen base LM + tokenizer from a registry preset or custom HF id. | | Loaders | Load Qwen-Scope SAE | Load one layer of a Qwen-Scope SAE pack (LRU-cached). | | Capture | Analyze Prompt (SAE) | Run a prompt → capture residual at the SAE's layer → encode features. | | Capture | Analyze CLIP (SAE) | Same, but for ComfyUI CLIP text encoders (FLUX.2's Qwen3-8B, Qwen-Image's Qwen2.5-VL, fp4mixed/fp8mixed/GGUF — anything ComfyUI can load). | | Capture | Capture Residual | Lower-level: just capture residual hidden states (no SAE). | | Inspect | Feature Heatmap | Top-K features × tokens, coloured by activation. → IMAGE | | Inspect | Token Activation Strip | Tokens as coloured chips by per-token max or a chosen feature. → IMAGE | | Inspect | Top Features | Bar chart + markdown table of top mean-activation features. → IMAGE + STRING | | Steer | Feature → Direction | Build a steer plan from an SAE feature's decoder vector. | | Steer | Residual Steer (custom dir) | Build a steer plan from any direction tensor. | | Steer | Steer Conditioning | Add a steer direction to CONDITIONING (post-CLIPTextEncode). Cleanest for image flows. | | Steer | Steer CLIP | Install hooks on a CLIP and return it; for multi-layer residual interventions. | | Steer | Clear CLIP Steering | Remove all QwenScope hooks from a CLIP. | | Lens | Save Lens | Persist a steer plan to lenses/qwenscope/<name>.safetensors. | | Lens | Load Lens | Load a saved lens (or any compatible .pt / .safetensors) as a steer plan. | | Lens | Combine Lenses | Sum up to 4 weighted steer plans into one (e.g. 0.8 × cinematic + 0.5 × moody). | | Train | Train Lens (Contrastive) | Train a direction from positive / negative prompt pairs (~30s). | | Generate | Generate (Qwen) | model.generate() with optional steering hooks active. |

Quick start graph

Load Qwen Model ─────────────────────┐
                                     ├──► Analyze ──► Feature Heatmap (IMAGE)
Load Qwen-Scope SAE ─────────────────┘                Token Strip      (IMAGE)
                                                      Top Features    (IMAGE)
                                                                       (STRING)

Load Qwen-Scope SAE ──► Feature → Direction ──► Generate (Qwen) ─► STRING
                         (feature_idx, strength,     ▲
                          layer_spec, normalize)     │
                                                     │
                       Load Qwen Model ──────────────┘

Image-model flows (FLUX.2 / Qwen-Image)

Image models that use a Qwen text encoder can be analysed and steered without loading a separate Qwen LLM. Two patterns:

Conditioning-level (preferred for most workflows — same shape as ConceptSteer):

[Load CLIP] ─► CLIP ─► [CLIPTextEncode] ─► CONDITIONING ─► [Steer Conditioning] ─► [KSampler]
                                                              ▲
                                  [Load Lens] / [Feature → Direction]

CLIP-level (multi-layer residual interventions):

[Load CLIP] ─► CLIP ─► [Steer CLIP] ─► CLIP ─► [CLIPTextEncode] ─► [KSampler]
                            ▲
              [Feature → Direction]

Conditioning-level steering runs once on the encoded tensor (no hooks, faster). CLIP-level steering can intervene at multiple layers but installs hooks that persist on the CLIP object until you run Clear CLIP Steering.

Training your own lens

For concepts that aren't a single SAE feature — "cinematic", "vintage film", "ethereal" — train a direction from prompt pairs. Same recipe as ConceptSteer's Train Lens (Contrastive), but routed through the QwenScope CLIP bridge so it works with FLUX.2 / Qwen-Image / quantised encoders without loading a separate Qwen LLM.

[Load CLIP] ─► CLIP ─► [Train Lens (Contrastive)] ─► steer ─► [Save Lens]
                              ▲
                  positive_texts (one per line)
                  negative_texts (one per line, equal count)
                  layer (-1 = last, recommended for Steer Conditioning)

Tips for good lenses:

• 8–12 prompt pairs is usually enough; 20+ is diminishing returns.
• Pairs should differ only on the target concept. "A cinematic dramatic portrait" vs "A casual snapshot portrait" beats "A cinematic dramatic portrait" vs "A photo of a dog."
• Use layer = -1 (last layer) for use with Steer Conditioning; pick a middle layer for Steer CLIP at the same layer.
• Set save_name to auto-persist after training; or chain into Save Lens manually.

Lenses

Save a steer plan to disk and reuse it across workflows. Same idea as ConceptSteer's lenses, format is .safetensors with metadata in the header.

[Feature → Direction] ─► steer ─► [Save Lens (name="cinematic")] ─► steer
                                                                       │
                                                                       ▼
                                                  later: [Load Lens] ─► steer ─► [Steer Conditioning]

Saved lenses land in lenses/qwenscope/. The dropdown in Load Lens also picks up .pt lenses from any lenses/ directory under either the package or ComfyUI/lenses/ (so your conceptsteer lenses still load).

Quantisation: the bridge works on whatever weights ComfyUI loaded. fp16/bf16 is drop-in. fp8mixed activations are close enough to bf16 that SAE features fire correctly. fp4mixed is noisier — high-mean features are still robust, rare-firing ones drift; expect to push strength higher than you would on full precision.

Pairings worth trying:

| Image model | Text encoder | SAE pack | | --- | --- | --- | | FLUX.2 [klein] 9B | Qwen3-8B | Qwen/SAE-Res-Qwen3-8B-Base-W64K-L0_50 (or L0_100) | | Qwen-Image | Qwen2.5-VL-7B | n/a yet (no Qwen2.5-VL SAE released) |

Layer spec syntax

Used by Feature → Direction and Residual Steer to choose which transformer layers receive the steering injection:

| Spec | Meaning | | --- | --- | | all | every layer, weight 1.0 | | early / middle / late | bottom / middle / top third of layers | | 5 | layer 5 only | | 5-8 | layers 5 through 8 | | 5,10-12,15:0.5 | mix of indices and ranges; :N overrides per-entry weight |

Architecture notes

• SAE storage: each pack ships as layer{n}.sae.pt files (one per transformer layer) holding W_enc, W_dec, b_enc, b_dec. The runtime LRU-caches up to 8 layer shards on the SAE device.
• Hook point: residual stream after model.model.layers[layer] — same convention as the Qwen-Scope explorer.
• Feature direction: taking column i of W_dec gives a d_model-dim vector pointing in the direction the SAE associates with feature i. Adding it back to the residual stream during generation is the basic steering recipe.
• Steering normalisation: when normalize=True, the direction is unit-normed and scaled by 0.3 × mean residual norm × strength, so strength is roughly comparable across layers and prompts.

Credits

• Qwen team for releasing Qwen-Scope and the SAE training code.

License

Apache-2.0.