diff --git a/README.md b/README.md index 6d62d5753..e076bfb78 100644 --- a/README.md +++ b/README.md @@ -99,6 +99,7 @@ API and command-line option may change frequently.*** - Control Net support with SD 1.5 - [ADetailer](./docs/models/adetailer.md) - LoRA support, same as [stable-diffusion-webui](https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Features#lora) + - [Textual inversion embeddings](./docs/embeddings.md) for SD 1.x/2.x/SDXL via `--embd-dir` and `` tags - Latent Consistency Models support (LCM/LCM-LoRA) - Faster and memory efficient latent decoding with [TAESD](./docs/models/taesd.md) - Upscale images generated with [ESRGAN](./docs/models/esrgan.md) diff --git a/docs/embeddings.md b/docs/embeddings.md new file mode 100644 index 000000000..a3d90bbc4 --- /dev/null +++ b/docs/embeddings.md @@ -0,0 +1,93 @@ +# Textual Inversion Embeddings + +Textual inversion embeddings add a new trigger word (or a few vectors) to the +CLIP text encoder. They are supported for **SD 1.x, SD 2.x and SDXL** +checkpoints, which use `FrozenCLIPEmbedderWithCustomWords`. Other families +(SD3, FLUX, Qwen Image, ...) do not use this code path and ignore `--embd-dir`. + +Supported weight files are `.safetensors`, `.pt`, `.ckpt` and `.gguf`. + +## Directory layout + +Point `--embd-dir` at the folder holding the embedding files. Subdirectories +are scanned recursively: + +```sh +./bin/sd-cli -m ../models/v1-5-pruned-emaonly.safetensors \ + --embd-dir ../models/embeddings \ + -p "ehartnoveau, girl playing the guitar" \ + --steps 20 -o output.png +``` + +The trigger word defaults to the file name without its extension. For +`embeddings/ehartnoveau.safetensors` the trigger word is `ehartnoveau`. +File names are matched case-insensitively, and a relative path without its +extension (for example `style/ehartnoveau` for +`embeddings/style/ehartnoveau.safetensors`) also works as a trigger word. + +When two files share the same stem, the first file found keeps the bare stem +and every file keeps its own relative path. Prefer unique stems or use the +relative path form to disambiguate. + +Run with `-v` to list the discovered embeddings at startup: + +```text +embedding_dir: "/path/to/embeddings", +embeddings: { + "ehartnoveau": "/path/to/embeddings/ehartnoveau.safetensors", + ... +} +``` + +## Using embeddings in prompts + +### 1. Bare trigger word + +Type the trigger word like any other word: + +```text +ehartnoveau, girl playing the guitar +``` + +Normal prompt weighting applies, so `(ehartnoveau:1.2)` boosts it and +`[ehartnoveau]` weakens it. + +### 2. Explicit `` tag + +When the file name is hard to type, contains spaces, or lives in a subdirectory, +reference it explicitly. This mirrors the `` syntax: + +```text +a lovely cat, +a lovely cat, +a lovely cat, +``` + +Rules: + +- The tag prefix is `embedding` or the short alias `emb`, case-insensitive: + ``, ``, `` all work. +- The name may be a file stem (`my_emb`), a relative path with forward or + backward slashes (`sub/dir/my_emb`), and may include the file extension + (`my_emb.safetensors`), which is ignored for the lookup. +- The weight is optional. `` means weight `1.0`. + With a weight, the tag expands to `(trigger:weight)` before the usual + `(text:weight)` parser runs, for example + `` becomes `(ehartnoveau:0.8)`. +- An unknown name logs a warning and falls back to the stripped name as plain + text, so a typo does not leak `` tokens into the tokenizer. +- If the file stem collides with an ordinary word (for example an embedding + named `cat`), rename the file to a unique stem instead of using a tag: the + tag expands to the same trigger word, so a bare `cat` elsewhere in the + prompt would still activate the embedding. + +## Troubleshooting + +- `embedding '...' not found`: the file is not under `--embd-dir`, has an + unsupported extension, or the name was mistyped. Check the verbose startup + listing for the exact key. +- No visible effect: make sure the checkpoint is SD 1.x, SD 2.x or SDXL. + Other models ignore textual inversion files. +- Weight has no effect and the log mentions `contains attention syntax`: + the trigger word itself contains `()[]:\`. Rename the file to a plain + stem and use the tag form. diff --git a/docs/models/sd.md b/docs/models/sd.md index 2d1ab53e3..197d72226 100644 --- a/docs/models/sd.md +++ b/docs/models/sd.md @@ -3,6 +3,8 @@ Stable Diffusion 1.x, 2.x, and SDXL support text-to-image and image-to-image generation from single-file checkpoints. +See [Textual Inversion Embeddings](../embeddings.md) for `--embd-dir` and `` usage with SD 1.x/2.x/SDXL. + ## Model files | Component | Weights and downloads | CLI option | Requirement | diff --git a/examples/common/common.cpp b/examples/common/common.cpp index 0c155c1b8..8db4ddabf 100644 --- a/examples/common/common.cpp +++ b/examples/common/common.cpp @@ -496,7 +496,7 @@ ArgOptions SDContextParams::get_options() { &motion_module_path}, {"", "--embd-dir", - "embeddings directory", + "textual inversion embeddings directory (SD 1.x/2.x/SDXL only, see docs/embeddings.md)", 0, &embedding_dir}, {"", @@ -813,12 +813,65 @@ void SDContextParams::build_embedding_map() { return; } - for (auto& p : fs::directory_iterator(embedding_dir_path)) { - if (!p.is_regular_file()) + auto to_lower = [](std::string s) { + std::transform(s.begin(), s.end(), s.begin(), [](unsigned char c) { + return static_cast(std::tolower(c)); + }); + return s; + }; + // Keys are matched case-insensitively downstream, so keep the first file + // when two names only differ by case or share the same file stem. + std::map lower_seen; + for (const auto& kv : embedding_map) { + std::string lower = to_lower(kv.first); + std::replace(lower.begin(), lower.end(), '\\', '/'); + lower_seen.emplace(lower, kv.first); + } + auto try_insert = [&](const std::string& key, const std::string& value) { + if (key.empty()) { + return; + } + std::string lower = to_lower(key); + std::replace(lower.begin(), lower.end(), '\\', '/'); + if (lower_seen.find(lower) != lower_seen.end()) { + return; + } + if (embedding_map.find(key) != embedding_map.end()) { + lower_seen[lower] = key; + return; + } + embedding_map[key] = value; + lower_seen[lower] = key; + }; + + // Collect first so relative-path keys can be registered before stem keys. + // This keeps a root-level file reachable when a nested file shares its + // stem: the root file owns the bare stem while the nested file stays + // reachable via its relative path. + std::vector> rel_entries; + std::vector> stem_entries; + std::error_code ec; + fs::recursive_directory_iterator end; + // skip_permission_denied keeps readable siblings discoverable when a + // subdirectory is inaccessible; other traversal errors are reported below. + fs::recursive_directory_iterator it(embedding_dir_path, fs::directory_options::skip_permission_denied, ec); + if (ec) { + LOG_WARN("failed to scan embedding dir '%s': %s", embedding_dir.c_str(), ec.message().c_str()); + return; + } + for (; it != end; it.increment(ec)) { + if (ec) { + LOG_WARN("error scanning embedding dir '%s': %s", embedding_dir.c_str(), ec.message().c_str()); + break; + } + const auto& p = *it; + std::error_code file_ec; + if (!p.is_regular_file(file_ec) || file_ec) { continue; + } auto path = p.path(); - std::string ext = path.extension().u8string(); + std::string ext = to_lower(path.extension().u8string()); bool valid = false; for (auto& e : valid_ext) { @@ -830,10 +883,24 @@ void SDContextParams::build_embedding_map() { if (!valid) continue; - std::string key = path.stem().u8string(); - std::string value = path.u8string(); - - embedding_map[key] = value; + std::string value = path.u8string(); + std::string stem_key = path.stem().u8string(); + std::string rel_key; + // Lexical only: never resolve symlinks, so the documented + // relative path stays selectable even for symlinked files. + fs::path rel = path.lexically_relative(embedding_dir_path); + if (!rel.empty()) { + rel.replace_extension(); + rel_key = rel.generic_u8string(); + } + rel_entries.emplace_back(rel_key, value); + stem_entries.emplace_back(stem_key, value); + } + for (const auto& entry : rel_entries) { + try_insert(entry.first, entry.second); + } + for (const auto& entry : stem_entries) { + try_insert(entry.first, entry.second); } } diff --git a/examples/server/README.md b/examples/server/README.md index 8673eb80c..c31795e25 100644 --- a/examples/server/README.md +++ b/examples/server/README.md @@ -336,7 +336,7 @@ steps/CFG for the chosen weights. Metadata inspection, conversion, standalone upscaling and ADetailer are CLI workflows rather than model-loading recipes. ESRGAN and LTX latent upscalers are selected through generation upscaling settings with `--hires-upscalers-dir`; -textual inversion uses the startup `--embd-dir` option. +textual inversion uses the startup `--embd-dir` option (see `docs/embeddings.md` for trigger words and `` tags). SVD is not available through the current server pipeline. diff --git a/src/conditioning/conditioner.hpp b/src/conditioning/conditioner.hpp index 808375c4e..a38ef36ec 100644 --- a/src/conditioning/conditioner.hpp +++ b/src/conditioning/conditioner.hpp @@ -1,12 +1,14 @@ #ifndef __SD_CONDITIONING_CONDITIONER_HPP__ #define __SD_CONDITIONING_CONDITIONER_HPP__ +#include #include #include #include #include #include #include +#include #include #include #include "conditioning/prompt_input.h" @@ -197,8 +199,18 @@ struct FrozenCLIPEmbedderWithCustomWords : public Conditioner { } for (const auto& kv : orig_embedding_map) { std::string name = normalize_embedding_name(kv.first); + std::replace(name.begin(), name.end(), '\\', '/'); embedding_map[name] = kv.second; tokenizer->add_special_token(name); + // Allow Windows-style separators in trigger words. + if (name.find('/') != std::string::npos) { + std::string backslash_name = name; + std::replace(backslash_name.begin(), backslash_name.end(), '/', '\\'); + if (embedding_map.find(backslash_name) == embedding_map.end()) { + embedding_map[backslash_name] = kv.second; + tokenizer->add_special_token(backslash_name); + } + } } bool force_clip_f32 = !embedding_map.empty(); if (sd_version_is_sd1(version)) { @@ -376,9 +388,285 @@ struct FrozenCLIPEmbedderWithCustomWords : public Conditioner { return name; } + // Resolve an name to a trigger word present in embedding_map. + // Accepts a bare file stem ("my_emb"), a relative path ("sub/dir/my_emb") + // with or without extension, and backslash separators. Returns the matched + // map key, or an empty string when nothing matches. + std::string find_embedding_trigger(const std::string& raw_name) const { + std::string norm = normalize_embedding_name(trim(raw_name)); + std::replace(norm.begin(), norm.end(), '\\', '/'); + if (norm.empty()) { + return {}; + } + auto lookup = [&](const std::string& key) -> std::string { + auto it = embedding_map.find(key); + return it != embedding_map.end() ? it->first : std::string(); + }; + if (auto hit = lookup(norm); !hit.empty()) { + return hit; + } + static const std::vector valid_ext = {".gguf", ".safetensors", ".pt", ".ckpt"}; + std::string no_ext = norm; + for (const auto& ext : valid_ext) { + if (no_ext.size() > ext.size() && no_ext.compare(no_ext.size() - ext.size(), ext.size(), ext) == 0) { + no_ext = no_ext.substr(0, no_ext.size() - ext.size()); + break; + } + } + if (no_ext != norm) { + if (auto hit = lookup(no_ext); !hit.empty()) { + return hit; + } + } + return {}; + } + + // Format a weight so parse_prompt_attention can read it back: decimal + // digits and '.' only, without exponent. The attention lexer accepts an + // optional sign followed by digits and dots, so "1e2" would be misread. + static std::string format_embedding_weight(float weight) { + std::ostringstream oss; + oss << std::setprecision(9) << std::noshowpoint << static_cast(weight); + std::string s = oss.str(); + bool has_exp = false; + for (char c : s) { + if (c == 'e' || c == 'E' || c == 'n' || c == 'N' || c == 'i' || c == 'I') { + has_exp = true; + break; + } + } + if (has_exp) { + oss.str(""); + oss.clear(); + oss << std::fixed << std::setprecision(10) << static_cast(weight); + s = oss.str(); + auto dot = s.find('.'); + if (dot != std::string::npos) { + while (!s.empty() && s.back() == '0') { + s.pop_back(); + } + if (!s.empty() && s.back() == '.') { + s.pop_back(); + } + if (s.empty() || s == "-" || s == "+") { + s = "0"; + } + } + } + return s; + } + + // Expand tags, replacing triggers that contain attention punctuation with + // placeholders that survive parse_prompt_attention as one chunk. The + // caller restores placeholders to the real triggers after parsing. + std::string expand_embedding_tags_with_placeholders( + const std::string& text, + std::vector>& placeholder_map) const { + placeholder_map.clear(); + if (text.find('<') == std::string::npos) { + return text; + } + std::string result; + result.reserve(text.size()); + size_t pos = 0; + size_t placeholder_idx = 0; + bool expanded = false; + auto is_tag_space = [](char c) { + return c == ' ' || c == '\t' || c == '\n' || c == '\r' || c == '\f' || c == '\v'; + }; + auto matches_tag = [](const std::string& s, size_t p, const char* lit) { + for (size_t i = 0; lit[i] != '\0'; ++i) { + if (p + i >= s.size()) { + return false; + } + if (std::tolower((unsigned char)s[p + i]) != (unsigned char)lit[i]) { + return false; + } + } + return true; + }; + auto make_placeholder = [&](std::string& out) { + out.clear(); + out.push_back('\x1f'); + out += "embph"; + out += std::to_string(placeholder_idx++); + out.push_back('\x1f'); + }; + while (true) { + size_t lt = text.find('<', pos); + if (lt == std::string::npos) { + result += text.substr(pos); + break; + } + size_t cur = lt + 1; + size_t spaces = 0; + while (cur < text.size() && is_tag_space(text[cur]) && spaces < 64) { + ++cur; + ++spaces; + } + if (spaces >= 64 && cur < text.size() && is_tag_space(text[cur])) { + result += text.substr(pos, lt - pos + 1); + pos = lt + 1; + continue; + } + size_t name_len = 0; + if (matches_tag(text, cur, "embedding")) { + name_len = 9; + } else if (matches_tag(text, cur, "emb")) { + name_len = 3; + } else { + result += text.substr(pos, lt - pos + 1); + pos = lt + 1; + continue; + } + cur += name_len; + spaces = 0; + while (cur < text.size() && is_tag_space(text[cur]) && spaces < 64) { + ++cur; + ++spaces; + } + if (spaces >= 64 && cur < text.size() && is_tag_space(text[cur])) { + result += text.substr(pos, lt - pos + 1); + pos = lt + 1; + continue; + } + if (cur >= text.size() || text[cur] != ':') { + result += text.substr(pos, lt - pos + 1); + pos = lt + 1; + continue; + } + size_t content_start = cur + 1; + size_t gt = text.find('>', content_start); + if (gt == std::string::npos) { + result += text.substr(pos); + pos = text.size(); + break; + } + if (gt == content_start) { + result += text.substr(pos, gt - pos + 1); + pos = gt + 1; + continue; + } + std::string content = text.substr(content_start, gt - content_start); + std::string raw_name; + std::string raw_weight; + std::string formatted_weight; + float weight = 1.0f; + bool has_weight = false; + auto colon = content.rfind(':'); + if (colon != std::string::npos) { + std::string maybe_name = trim(content.substr(0, colon)); + std::string maybe_weight = trim(content.substr(colon + 1)); + float parsed = 1.0f; + if (!maybe_name.empty() && parse_strict_float(maybe_weight, parsed) && std::isfinite(parsed)) { + raw_name = maybe_name; + raw_weight = maybe_weight; + formatted_weight = format_embedding_weight(parsed); + weight = parsed; + has_weight = true; + } else { + raw_name = trim(content); + } + } else { + raw_name = trim(content); + } + if (raw_name.empty()) { + LOG_WARN("ignoring tag with empty name"); + result += text.substr(pos, lt - pos); + pos = gt + 1; + expanded = true; + continue; + } + std::string trigger = find_embedding_trigger(raw_name); + bool known = !trigger.empty(); + if (!known) { + LOG_WARN("embedding '%s' not found, using '%s' as plain text", raw_name.c_str(), raw_name.c_str()); + std::string fallback = raw_name; + std::replace(fallback.begin(), fallback.end(), '\\', '/'); + auto slash = fallback.find_last_of('/'); + if (slash != std::string::npos && slash + 1 < fallback.size()) { + fallback = fallback.substr(slash + 1); + } + static const std::vector valid_ext = {".gguf", ".safetensors", ".pt", ".ckpt"}; + std::string lowered_fallback = normalize_embedding_name(fallback); + for (const auto& ext : valid_ext) { + if (lowered_fallback.size() > ext.size() && lowered_fallback.compare(lowered_fallback.size() - ext.size(), ext.size(), ext) == 0) { + fallback = fallback.substr(0, fallback.size() - ext.size()); + break; + } + } + fallback = trim(fallback); + trigger = fallback.empty() ? raw_name : fallback; + } + std::string replacement; + bool has_attention_syntax = trigger.find_first_of("()[]:\\") != std::string::npos; + bool needs_placeholder = known && has_attention_syntax; + if (has_weight && std::fabs(weight - 1.0f) > 1e-6f) { + if (has_attention_syntax) { + LOG_WARN("embedding '%s' contains attention syntax, ignoring weight %s", trigger.c_str(), raw_weight.c_str()); + if (needs_placeholder) { + make_placeholder(replacement); + placeholder_map.emplace_back(replacement, trigger); + } else { + replacement = trigger; + } + } else { + replacement = "(" + trigger + ":" + formatted_weight + ")"; + } + } else { + if (needs_placeholder) { + make_placeholder(replacement); + placeholder_map.emplace_back(replacement, trigger); + } else { + replacement = trigger; + } + } + result += text.substr(pos, lt - pos); + result += replacement; + pos = gt + 1; + expanded = true; + } + if (!expanded) { + return text; + } + return result; + } + + static void restore_embedding_placeholders( + std::string& s, + const std::vector>& placeholder_map) { + for (const auto& entry : placeholder_map) { + size_t p = 0; + while ((p = s.find(entry.first, p)) != std::string::npos) { + s.replace(p, entry.first.size(), entry.second); + p += entry.second.size(); + } + } + } + + // Expand "" and "" ("" alias, + // case-insensitive tag) into a trigger word that the textual-inversion path + // understands. A non-1.0 weight becomes "(trigger:weight)" so the existing + // attention parser applies it. Unknown names fall back to the stripped name + // so the prompt stays readable instead of leaking "" tokens. + std::string expand_embedding_tags(const std::string& text) const { + if (text.find('<') == std::string::npos) { + return text; + } + std::vector> placeholder_map; + std::string expanded = expand_embedding_tags_with_placeholders(text, placeholder_map); + if (expanded == text) { + return text; + } + restore_embedding_placeholders(expanded, placeholder_map); + LOG_VERBOSE("expanded embedding tags '%s' to '%s'", escape_newlines(text).c_str(), escape_newlines(expanded).c_str()); + return expanded; + } + bool append_embedding_tokens(std::string str, std::vector& bpe_tokens) { std::string name = normalize_embedding_name(std::move(str)); - auto iter = embedding_map.find(name); + std::replace(name.begin(), name.end(), '\\', '/'); + auto iter = embedding_map.find(name); if (iter == embedding_map.end()) { return false; } @@ -400,7 +688,14 @@ struct FrozenCLIPEmbedderWithCustomWords : public Conditioner { size_t min_length = 0, size_t max_length = 0, bool allow_overflow_expand = true) { - auto parsed_attention = parse_prompt_attention(text); + std::vector> placeholder_map; + std::string expanded_text = expand_embedding_tags_with_placeholders(text, placeholder_map); + auto parsed_attention = parse_prompt_attention(expanded_text); + for (auto& item : parsed_attention) { + restore_embedding_placeholders(item.first, placeholder_map); + } + text = expanded_text; + restore_embedding_placeholders(text, placeholder_map); { std::stringstream ss;