Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,7 @@ API and command-line option may change frequently.***
- Control Net support with SD 1.5
- [ADetailer](./docs/models/adetailer.md)
- LoRA support, same as [stable-diffusion-webui](https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Features#lora)
- [Textual inversion embeddings](./docs/embeddings.md) for SD 1.x/2.x/SDXL via `--embd-dir` and `<embedding:...>` tags
- Latent Consistency Models support (LCM/LCM-LoRA)
- Faster and memory efficient latent decoding with [TAESD](./docs/models/taesd.md)
- Upscale images generated with [ESRGAN](./docs/models/esrgan.md)
Expand Down
93 changes: 93 additions & 0 deletions docs/embeddings.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
# Textual Inversion Embeddings

Textual inversion embeddings add a new trigger word (or a few vectors) to the
CLIP text encoder. They are supported for **SD 1.x, SD 2.x and SDXL**
checkpoints, which use `FrozenCLIPEmbedderWithCustomWords`. Other families
(SD3, FLUX, Qwen Image, ...) do not use this code path and ignore `--embd-dir`.

Supported weight files are `.safetensors`, `.pt`, `.ckpt` and `.gguf`.

## Directory layout

Point `--embd-dir` at the folder holding the embedding files. Subdirectories
are scanned recursively:

```sh
./bin/sd-cli -m ../models/v1-5-pruned-emaonly.safetensors \
--embd-dir ../models/embeddings \
-p "ehartnoveau, girl playing the guitar" \
--steps 20 -o output.png
```

The trigger word defaults to the file name without its extension. For
`embeddings/ehartnoveau.safetensors` the trigger word is `ehartnoveau`.
File names are matched case-insensitively, and a relative path without its
extension (for example `style/ehartnoveau` for
`embeddings/style/ehartnoveau.safetensors`) also works as a trigger word.

When two files share the same stem, the first file found keeps the bare stem
and every file keeps its own relative path. Prefer unique stems or use the
relative path form to disambiguate.

Run with `-v` to list the discovered embeddings at startup:

```text
embedding_dir: "/path/to/embeddings",
embeddings: {
"ehartnoveau": "/path/to/embeddings/ehartnoveau.safetensors",
...
}
```

## Using embeddings in prompts

### 1. Bare trigger word

Type the trigger word like any other word:

```text
ehartnoveau, girl playing the guitar
```

Normal prompt weighting applies, so `(ehartnoveau:1.2)` boosts it and
`[ehartnoveau]` weakens it.

### 2. Explicit `<embedding:...>` tag

When the file name is hard to type, contains spaces, or lives in a subdirectory,
reference it explicitly. This mirrors the `<lora:...>` syntax:

```text
a lovely cat, <embedding:ehartnoveau:0.8>
a lovely cat, <embedding:style/ehartnoveau:0.8>
a lovely cat, <emb:my embedding>
```

Rules:

- The tag prefix is `embedding` or the short alias `emb`, case-insensitive:
`<embedding:...>`, `<Embedding:...>`, `<emb:...>` all work.
- The name may be a file stem (`my_emb`), a relative path with forward or
backward slashes (`sub/dir/my_emb`), and may include the file extension
(`my_emb.safetensors`), which is ignored for the lookup.
- The weight is optional. `<embedding:my_emb>` means weight `1.0`.
With a weight, the tag expands to `(trigger:weight)` before the usual
`(text:weight)` parser runs, for example
`<embedding:ehartnoveau:0.8>` becomes `(ehartnoveau:0.8)`.
- An unknown name logs a warning and falls back to the stripped name as plain
text, so a typo does not leak `<embedding:...>` tokens into the tokenizer.
- If the file stem collides with an ordinary word (for example an embedding
named `cat`), rename the file to a unique stem instead of using a tag: the
tag expands to the same trigger word, so a bare `cat` elsewhere in the
prompt would still activate the embedding.

## Troubleshooting

- `embedding '...' not found`: the file is not under `--embd-dir`, has an
unsupported extension, or the name was mistyped. Check the verbose startup
listing for the exact key.
- No visible effect: make sure the checkpoint is SD 1.x, SD 2.x or SDXL.
Other models ignore textual inversion files.
- Weight has no effect and the log mentions `contains attention syntax`:
the trigger word itself contains `()[]:\`. Rename the file to a plain
stem and use the tag form.
2 changes: 2 additions & 0 deletions docs/models/sd.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@
Stable Diffusion 1.x, 2.x, and SDXL support text-to-image and image-to-image
generation from single-file checkpoints.

See [Textual Inversion Embeddings](../embeddings.md) for `--embd-dir` and `<embedding:...>` usage with SD 1.x/2.x/SDXL.

## Model files

| Component | Weights and downloads | CLI option | Requirement |
Expand Down
83 changes: 75 additions & 8 deletions examples/common/common.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -496,7 +496,7 @@ ArgOptions SDContextParams::get_options() {
&motion_module_path},
{"",
"--embd-dir",
"embeddings directory",
"textual inversion embeddings directory (SD 1.x/2.x/SDXL only, see docs/embeddings.md)",
0,
&embedding_dir},
{"",
Expand Down Expand Up @@ -813,12 +813,65 @@ void SDContextParams::build_embedding_map() {
return;
}

for (auto& p : fs::directory_iterator(embedding_dir_path)) {
if (!p.is_regular_file())
auto to_lower = [](std::string s) {
std::transform(s.begin(), s.end(), s.begin(), [](unsigned char c) {
return static_cast<char>(std::tolower(c));
});
return s;
};
// Keys are matched case-insensitively downstream, so keep the first file
// when two names only differ by case or share the same file stem.
std::map<std::string, std::string> lower_seen;
for (const auto& kv : embedding_map) {
std::string lower = to_lower(kv.first);
std::replace(lower.begin(), lower.end(), '\\', '/');
lower_seen.emplace(lower, kv.first);
}
auto try_insert = [&](const std::string& key, const std::string& value) {
if (key.empty()) {
return;
}
std::string lower = to_lower(key);
std::replace(lower.begin(), lower.end(), '\\', '/');
if (lower_seen.find(lower) != lower_seen.end()) {
return;
}
if (embedding_map.find(key) != embedding_map.end()) {
lower_seen[lower] = key;
return;
}
embedding_map[key] = value;
lower_seen[lower] = key;
};

// Collect first so relative-path keys can be registered before stem keys.
// This keeps a root-level file reachable when a nested file shares its
// stem: the root file owns the bare stem while the nested file stays
// reachable via its relative path.
std::vector<std::pair<std::string, std::string>> rel_entries;
std::vector<std::pair<std::string, std::string>> stem_entries;
std::error_code ec;
fs::recursive_directory_iterator end;
// skip_permission_denied keeps readable siblings discoverable when a
// subdirectory is inaccessible; other traversal errors are reported below.
fs::recursive_directory_iterator it(embedding_dir_path, fs::directory_options::skip_permission_denied, ec);
if (ec) {
LOG_WARN("failed to scan embedding dir '%s': %s", embedding_dir.c_str(), ec.message().c_str());
return;
}
for (; it != end; it.increment(ec)) {
if (ec) {
LOG_WARN("error scanning embedding dir '%s': %s", embedding_dir.c_str(), ec.message().c_str());
break;
}
const auto& p = *it;
std::error_code file_ec;
if (!p.is_regular_file(file_ec) || file_ec) {
continue;
}

auto path = p.path();
std::string ext = path.extension().u8string();
std::string ext = to_lower(path.extension().u8string());

bool valid = false;
for (auto& e : valid_ext) {
Expand All @@ -830,10 +883,24 @@ void SDContextParams::build_embedding_map() {
if (!valid)
continue;

std::string key = path.stem().u8string();
std::string value = path.u8string();

embedding_map[key] = value;
std::string value = path.u8string();
std::string stem_key = path.stem().u8string();
std::string rel_key;
// Lexical only: never resolve symlinks, so the documented
// relative path stays selectable even for symlinked files.
fs::path rel = path.lexically_relative(embedding_dir_path);
if (!rel.empty()) {
rel.replace_extension();
rel_key = rel.generic_u8string();
}
rel_entries.emplace_back(rel_key, value);
stem_entries.emplace_back(stem_key, value);
}
for (const auto& entry : rel_entries) {
try_insert(entry.first, entry.second);
}
for (const auto& entry : stem_entries) {
try_insert(entry.first, entry.second);
}
}

Expand Down
2 changes: 1 addition & 1 deletion examples/server/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -336,7 +336,7 @@ steps/CFG for the chosen weights.
Metadata inspection, conversion, standalone upscaling and ADetailer are CLI
workflows rather than model-loading recipes. ESRGAN and LTX latent upscalers
are selected through generation upscaling settings with `--hires-upscalers-dir`;
textual inversion uses the startup `--embd-dir` option.
textual inversion uses the startup `--embd-dir` option (see `docs/embeddings.md` for trigger words and `<embedding:...>` tags).

SVD is not available through the current server pipeline.

Expand Down
Loading
Loading