Skip to content

linux: do the hash multiplies in unsigned - #397

Merged
jiegec merged 1 commit into
lsof-org:masterfrom
Ar-maan05:linux-hash-unsigned
Nov 17, 2026
Merged

jiegec merged 1 commit into
lsof-org:masterfrom
Ar-maan05:linux-hash-unsigned

Conversation

@Ar-maan05

Copy link
Copy Markdown
Contributor

The linux hash macros multiply by 31415 in int, so they overflow for basically any real pid, inode or ipv6 address. On a UBSan build plain lsof -n -P -E on master prints:

lib/dialects/linux/dnode.c:629:13: runtime error: signed integer overflow: 1279995 * 31415 cannot be represented in type 'int'
lib/dialects/linux/dnode.c:72:9: runtime error: signed integer overflow: 599593 * 31415 cannot be represented in type 'int'
lib/dialects/linux/dfile.c:135:34: runtime error: signed integer overflow: 58720318 * 31415 cannot be represented in type 'int'
lib/dialects/linux/dsock.c:3026:25: runtime error: signed integer overflow: ...

the bucket mask hides it on normal builds so nothing visibly breaks, its just UB. This makes HASHPID, HASHPINFO, SFHASHDEVINO/SFHASHRDEVI and the IPv6 IPC hash multiply in unsigned. Low bits are the same so every bucket stays the same.

Tested: clang -fsanitize=undefined build, lsof -n -P -E and lsof on a few file names are clean now (7 overflow sites before), make check passes (39 pass, 3 skip).

Comment thread lib/dialects/linux/dfile.c Outdated
Comment on lines 54 to 58
#define SFHASHDEVINO(maj, min, ino, mod) \
((int)(((int)((((int)(maj + 1)) * ((int)((min + 1)))) + ino) * 31415) & \
((int)(((int)((((unsigned int)(maj + 1)) * ((unsigned int)((min + 1)))) + \
ino) * \
31415u) & \
(mod - 1)))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we keep the calculation unsigned until after masking?

Suggested change
#define SFHASHDEVINO(maj, min, ino, mod) \
((int)(((int)((((int)(maj + 1)) * ((int)((min + 1)))) + ino) * 31415) & \
((int)(((int)((((unsigned int)(maj + 1)) * ((unsigned int)((min + 1)))) + \
ino) * \
31415u) & \
(mod - 1)))
#define SFHASHDEVINO(maj, min, ino, mod) \
((int)(((((unsigned int)(maj) + 1u) * \
((unsigned int)(min) + 1u) + (unsigned int)(ino)) * \
31415u) & ((mod) - 1)))

The same could apply to SFHASHRDEVI below.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good call, applied it to both SFHASHDEVINO and SFHASHRDEVI so its unsigned all the way until the mask now. buckets come out the same, ubsan build is still clean

HASHPID, HASHPINFO, SFHASHDEVINO, SFHASHRDEVI and the IPv6 IPC hash multiplied in int, so any PID above 68356, most inode and device numbers, and most IPv6 addresses overflowed a signed int, which is undefined behaviour. The bucket mask hides it on usual builds, but UBSan reports it on every run and lsof's own tests hit it hundreds of times. Do the multiplies in unsigned int instead. The low bits are the same as before, so every value still lands in the same bucket.
@jiegec
jiegec merged commit c7c4c50 into lsof-org:master Nov 17, 2026
17 checks passed
@Ar-maan05
Ar-maan05 deleted the linux-hash-unsigned branch November 17, 2026 12:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants