Skip to content

ext/mbstring: Faster UTF-8 validation following simdutf - #25473

Open
lemire wants to merge 3 commits into
php:masterfrom
lemire:mbstring-utf8-lookup
Open

lemire wants to merge 3 commits into
php:masterfrom
lemire:mbstring-utf8-lookup

Conversation

@lemire

@lemire lemire commented Nov 15, 2026 •

Copy link
Copy Markdown

This PR updates the SIMD UTF-8 validation behind mb_check_encoding(), and behind the functions that validate UTF-8 before taking a fast path (mb_strlen(), mb_substr(), ...). It uses the "lookup" algorithm from

John Keiser and Daniel Lemire, Validating UTF-8 In Less Than One Instruction Per Byte, Software: Practice and Experience 51 (5), 2021. https://doi.org/10.1002/spe.2940

as implemented in simdutf. The simdutf library is used by mainstream browsers and major JavaScript runtimes. It is thus quite solid (used in production for many years).

Changes

  • AVX2: the existing kernel used part of this algorithm, but checked byte structure separately and tested for errors every 32 bytes. It now uses the full algorithm (three nibble lookups plus the check of the 2nd and 3rd previous bytes) and accumulates errors over blocks. There is also an ASCII fast path for 64-byte blocks. The tail and short strings use overlapping loads instead of a copy. Short strings may read up to 16 bytes of the zend_string header before the content, which the old code already did, and those bytes are cleared.
  • AVX-512 (new): a 64-byte kernel (AVX-512 F/BW/VL/DQ/CD) with a masked load for the tail. It is selected at runtime, via ifunc or a function pointer, like the AVX2 kernel. It is enabled on x86-64 only, and only when the AVX2 kernel is also selected at runtime, which follows ext/standard/base64.c. For long strings it checks one unaligned block, then continues from the next 64-byte boundary. The three bytes of context for that boundary are read from inside the string.
  • ARM64: a NEON implementation of the same algorithm (vqtbl1q_u8) replaces the SSE2 code emulated through zend_simd.h.
  • SSE2 and scalar fallbacks: unchanged.

Performance

mb_check_encoding($s, 'UTF-8'), ns per call; the string is modified before each call so the cached validity flag is cleared.

Intel Xeon Gold 6338 (Ice Lake), GCC 16:

input master AVX2 AVX-512
ASCII, 16 B 37.6 23.9 23.7
ASCII, 1 KiB 55.7 34.3 33.2
ASCII, 1 MiB 30,970 20,862 12,151
UTF-8 (CJK), 258 B 60.1 45.1 38.0
UTF-8 (CJK), 1 KiB 170 104 81
UTF-8 (CJK), 1 MiB 150,212 81,897 59,083

Apple M4 Max, Apple clang:

input master NEON
ASCII, 16 B 14.4 14.1
ASCII, 1 KiB 43.2 19.9
ASCII, 1 MiB 32,410 9,820
UTF-8 (CJK), 258 B 38.3 23.6
UTF-8 (CJK), 1 KiB 143 76
UTF-8 (CJK), 1 MiB 143,200 75,802

None of the measured sizes (16 B to 1 MiB, ASCII, Latin, CJK, emoji) got slower.

Testing

  • New test ext/mbstring/tests/mb_check_encoding_utf8_blocks.phpt. It puts valid, invalid and truncated sequences at every offset of strings up to 140 bytes, and compares against PCRE's UTF-8 validation. The string passed to mb_check_encoding() is a pack() copy, so the UTF-8 flag set by preg_match('//u') is clear and the SIMD path actually runs. It also checks strings of 2048 bytes and longer, which is when the AVX-512 kernel aligns its loads.
  • Differential fuzzing of each kernel against a scalar reference and simdutf. Strings were placed against inaccessible pages to catch out-of-bounds reads.
  • Random strings compared against preg_match('//u', ...).
  • mbstring test suite with:
    • AVX-512 via ifunc
    • AVX2 only
    • clang + ASan with function-pointer dispatch
    • aarch64 Linux under qemu
    • macOS arm64

#24362 also changes the AVX2 dispatch code in mbstring.c, for i386 clang; whichever lands second will need a small rebase.

Use the "lookup" algorithm of Keiser and Lemire, "Validating UTF-8 In
Less Than One Instruction Per Byte", Software: Practice and Experience
51 (5), 2021, as implemented in the simdutf library, for
mb_check_encoding() and the other functions which validate UTF-8:

- AVX2: full lookup algorithm (the previous version only adopted part of
  it), errors accumulated over blocks, 64-byte ASCII fast path, and no
  copying for the tail of the string or for short strings.
- AVX-512 (new): selected at runtime when available, on x86-64.
- ARM64: NEON implementation of the same algorithm, replacing the
  emulated SSE2 code.

Long strings are validated about twice as fast (UTF-8 text) to three
times as fast (ASCII); short strings are not slower.
The new test called preg_match('//u') before mb_check_encoding().
That sets IS_STR_VALID_UTF8, and str_repeat($s, 1) returns the same
string, so valid input never reached the new kernels. Copy the
bytes with pack() first, and cover strings longer than 2048 bytes.

The AVX-512 alignment prologue loaded context through a pointer
before the zend_string. Read the three bytes from inside the string.
OR the NEON continuation checks, matching AVX2.

Record the speedup in NEWS.
@lemire

lemire commented Nov 15, 2026

Copy link
Copy Markdown
Author

Opened by mistake. Closing.

@lemire lemire closed this Nov 15, 2026
_mm512_setr_epi8 does not exist in GCC or clang. Go back to the
_mm512_setr_epi64 constant, which has the same bytes.
@lemire lemire reopened this Nov 15, 2026
@lemire

lemire commented Nov 15, 2026

Copy link
Copy Markdown
Author

reopening, as it is now ready for review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant