You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
fix(lora): cast the LoRA diff to the weight's type where the backend can't add F32 to F16 - #2383
Merging a LoRA into F16 weights (--lora-apply-mode immediately) aborts on Metal:
ggml/src/ggml-backend.cpp:1368: pre-allocated tensor (cond_stage_model.transformer.text_model.encoder.layers.0.mlp.fc1.weight (view)) in a buffer (MTL0) that cannot run the operation (ADD)
LoraModel::build_lora_graph adds the F32 diff to each F16 weight in place: ADD(F16, F32). Metal's supports_op accepts ADD only when both sources have the same type, so ggml_backend_sched can't place the op on the backend that owns the weight's buffer.
The fix: the F32/F16 branch builds the add as before. If the diff's type differs from the weight's and ggml_backend_supports_op(runtime_backend, add) returns false, it builds ggml_cast(diff, weight type) and the add again. As asked in the review on #2380, the cast happens only where it is needed. Backends that can run ADD(F16, F32) keep the F32 diff and build the same graph as before. A capability check is used instead of a backend name, so any other backend with the same limit is covered too. Quantized and F32 weights are unchanged.
Command, the same on every backend (Illustrious-XL-v2.0, F16 SDXL, + Detail Tweaker XL):
sd-cli -m Illustrious-XL-v2.0.safetensors -W 512 -H 512 --steps 1 --seed 42 --lora-apply-mode immediately --lora-model-dir <loras> -p "a lighthouse on a cliff, <lora:detail-tweaker-xl:1.0>"
Sources: master 532d2c1, and the same tree with this change. Release builds. immediately is set explicitly: with auto, CUDA on an 8 GB card keeps the weights in RAM and picks at_runtime, which doesn't run this code.
Test machines:
Machine
CPU
RAM
GPU
OS
Toolchain
Mac
Apple M4 Max, 16 cores (12 performance + 4 efficiency)
64 GB unified
M4 Max 40-core GPU (Metal 4)
macOS 27.0.1
Apple clang 21.0.0, CMake 4.4.4
Windows PC
AMD Ryzen 5 3600, 6 cores / 12 threads
32 GB DDR4-2133
NVIDIA GeForce RTX 2060 SUPER 8 GB, driver 591.74
Windows 10 Pro 22H2 (19045)
MSVC 19.44 (VS 2022), Ninja, CMake 3.29; CUDA 12.9.86 (-DCMAKE_CUDA_ARCHITECTURES=75); Vulkan loader/headers 1.4.363 + shaderc (glslc) from vcpkg
Platform
Backend
master
this PR
Output master vs PR
Mac
Metal
aborts (above)
ok
-
Mac
CPU (-DGGML_METAL=OFF, -t 10)
ok
ok
byte-identical
Windows PC
CUDA
ok
ok
byte-identical
same, TE weights in VRAM (--params-backend te=cuda0)
CUDA
ok
ok
byte-identical
Windows PC
Vulkan
ok
ok
byte-identical
same, TE weights in VRAM (--params-backend te=vulkan0)
Vulkan
ok
ok
byte-identical
Windows PC
CPU
ok
ok
byte-identical
CUDA, Vulkan and CPU. The output is byte-identical to master, so no cast happens there. The only extra work is one supports_op call per merged weight.
Metal. The output is byte-identical to an earlier version of this branch that always cast.
LoRA applied. On every backend the image differs from the same prompt and seed without the LoRA.
Merge time. There is no measurable difference on any backend. Medians differ by under 1%, inside the spread between runs of the same build. Merge time was measured locally with a log line around LoraModel::apply's compute; the line is not part of this PR. Numbers are per run, summed over all modules: median (min-max).
Backend
master
this PR
Runs each
CUDA
1689 ms (1648-1690)
1679 ms (1655-1695)
3
Vulkan
1732 ms (1726-1749)
1734 ms (1726-1806)
5
CPU, Windows
13478 ms (13371-13582)
13560 ms (13460-13679)
3
CPU, macOS (-t 10, order alternated)
3352 ms (3328-3937)
3334 ms (3288-3394)
8
Metal, process CPU time; vs. always casting, since master aborts
786 ms (770-803)
785 ms (748-875)
5
Not tested: ROCm/HIP, SYCL, OpenCL, MUSA, Linux.
Formatting checked with clang-format 23.1.5 and the repository's .clang-format: no changes.
I used AI tools to help implement and test this change and draft this description. I reviewed the code, test results, and description.
If this PR adds or changes model support, I have added or updated its usage guide and relevant model index/README entries following the model documentation requirements. (Not applicable: no model support changes.)
No actionable comments were generated in the recent review.
Walkthrough
The LoRA graph now checks backend support for an addition with mixed tensor types. If unsupported, it casts the LoRA diff to the model weight type and rebuilds the addition.
This change lets LoRA merging complete on backends such as Metal that cannot add an F16 weight to an F32 diff. Behavior on backends that support the add is unchanged. No concrete merge-blocking risk was found.
Pre-merge checks: 4
Check name
Status
Explanation
Title check
The title names the main change: casting the LoRA diff to the weight's type when the backend cannot add F32 to F16. It matches the raw summary and the PR objectives. The title uses a conventional comm…
Linked Issues check
Objective [#2380]: merging a LoRA into F16 weights must not abort on Metal when the diff is F32. The diff at src/model/adapter/lora.hpp (else branch of build_lora_graph) builds the in-place ADD(model_…
Out of Scope Changes check
The whole-PR diff against merge base 532d2c1 changes only src/model/adapter/lora.hpp, adding 7 lines in the LoRA graph build for the F16/F32 add path. Every added line supports the objective in #2380:…
Description check
The description is thorough and covers every template section. The Summary explains the Metal abort, the root cause (ADD(F16, F32) is unsupported on Metal), and the conditional cast fix. Related Issue…
Full details: Title check
Explanation
The title names the main change: casting the LoRA diff to the weight's type when the backend cannot add F32 to F16. It matches the raw summary and the PR objectives. The title uses a conventional commit prefix and is specific. It is 89 characters, which is slightly long but acceptable.
Full details: Linked Issues check
Explanation
Objective [#2380]: merging a LoRA into F16 weights must not abort on Metal when the diff is F32. The diff at src/model/adapter/lora.hpp (else branch of build_lora_graph) builds the in-place ADD(model_tensor, diff) as before. When diff->type differs from model_tensor->type and ggml_backend_supports_op(runtime_backend, final_tensor) returns false, the change casts diff to the weight type with ggml_cast and rebuilds the add. This removes the unsupported ADD(F16, F32) on Metal, which the issue identifies as the cause. F32 and quantized weights keep the original path when the types match. The issue's proposed fix casts unconditionally; the PR narrows this to backends that refuse the mixed-type add, as the issue allows for precision reasons. The issue asks for no automated tests and no non-coding work. The Metal run was reported in the PR description, not verified here. Compilation of runtime_backend in this scope was not independently verified.
Full details: Out of Scope Changes check
Explanation
The whole-PR diff against merge base 532d2c1 changes only src/model/adapter/lora.hpp, adding 7 lines in the LoRA graph build for the F16/F32 add path. Every added line supports the objective in #2380: the type check, the supports_op check, the cast, and the rebuilt add. The comments explain the reason for the change. No unrelated files or refactoring appear in the diff. The benchmark tables and test notes in the PR description are not code changes.
Full details: Description check
Explanation
The description is thorough and covers every template section. The Summary explains the Metal abort, the root cause (ADD(F16, F32) is unsupported on Metal), and the conditional cast fix. Related Issue links #2380 and the review comment. Additional Information gives the reproduction command, test machines, a backend results table, byte-identical output checks, merge-time measurements, and the list of untested platforms. The checklist is completed, and the model-documentation item is marked not applicable. The template's required sections are all present and substantively filled in.
Finishing Touches
Generate unit tests (beta)
Create a new PR
Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
File: No configuration file from this repository applied
Commit: Unavailable
Layers: Repository UI
Review profile: CHILL
Plan: Advanced
Run ID: c194060b-c4a0-4ea9-a6c2-03ddd5651cda
Commits
Reviewing files that changed from the base of the PR and between 532d2c1 and 10b3c41.
Files selected for processing: 1
src/model/adapter/lora.hpp
Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 7 remain after this review.
…can't add F32 to F16
Merging a LoRA into F16 weights on Metal aborts: the in-place
ADD(F16, F32) can't run on MTL0, because Metal adds only tensors of the
same type. When the backend doesn't support that ADD, cast the diff to
the weight's type first. Backends that support it keep the F32 diff.
Fixesleejet#2380
The reason will be displayed to describe this comment to others. Learn more.
Could we cast the weight to F32 here, do the add, and use ggml_cpy to write it back as F16? I tried 1.0 + (-0.9999) in a small ggml test, and casting the delta first gave zero even though the sum fits in F16. The backend check looks good to me.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Merging a LoRA into F16 weights (
--lora-apply-mode immediately) aborts on Metal:LoraModel::build_lora_graphadds the F32 diff to each F16 weight in place:ADD(F16, F32). Metal'ssupports_opaccepts ADD only when both sources have the same type, soggml_backend_schedcan't place the op on the backend that owns the weight's buffer.The fix: the F32/F16 branch builds the add as before. If the diff's type differs from the weight's and
ggml_backend_supports_op(runtime_backend, add)returns false, it buildsggml_cast(diff, weight type)and the add again. As asked in the review on #2380, the cast happens only where it is needed. Backends that can runADD(F16, F32)keep the F32 diff and build the same graph as before. A capability check is used instead of a backend name, so any other backend with the same limit is covered too. Quantized and F32 weights are unchanged.Related Issue / Discussion
Fixes #2380. Review comment asking for this approach: #2380 (comment)
Additional Information
Command, the same on every backend (Illustrious-XL-v2.0, F16 SDXL, + Detail Tweaker XL):
Sources: master
532d2c1, and the same tree with this change. Release builds.immediatelyis set explicitly: withauto, CUDA on an 8 GB card keeps the weights in RAM and picksat_runtime, which doesn't run this code.Test machines:
-DCMAKE_CUDA_ARCHITECTURES=75); Vulkan loader/headers 1.4.363 + shaderc (glslc) from vcpkg-DGGML_METAL=OFF,-t 10)--params-backend te=cuda0)--params-backend te=vulkan0)supports_opcall per merged weight.Merge time. There is no measurable difference on any backend. Medians differ by under 1%, inside the spread between runs of the same build. Merge time was measured locally with a log line around
LoraModel::apply's compute; the line is not part of this PR. Numbers are per run, summed over all modules: median (min-max).-t 10, order alternated)Not tested: ROCm/HIP, SYCL, OpenCL, MUSA, Linux.
Formatting checked with clang-format 23.1.5 and the repository's
.clang-format: no changes.I used AI tools to help implement and test this change and draft this description. I reviewed the code, test results, and description.
Checklist