[vm] Avoid UB in FinalizeHash(hash)

FinalizeHash(hash) was trying to avoid UB in expression 1 << 32
by casting 1 to uintptr_t. This type however is not wide enough
on 32-bit platforms.

Instead just use explicit comparison hashbits < kBitsPerInt32 to
avoid overflow in left shift.

This bug went unnoticed for a while because it the only place
where we call FinalizeHash(hash) is in the snapshot profile
writer code and it only triggers when gen_snapshot is a
32-bit binary - which is only true on Windows, as Mac and Linux
seem to use simarm_x64 configuration instead.

This UB was explicitly affecting the code behavior because C++
compiler would either optimize out or change behavior of any
code that consumed value produced by FinalizeHash(hash).

Fixes https://github.com/flutter/flutter/issues/97764

TEST=vm/cc/DirectChainedHashMap

Change-Id: I39f2b09e7516c875b765e5a065d1c1331f89fa33
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/250741
Commit-Queue: Slava Egorov <vegorov@google.com>
Reviewed-by: Daco Harkes <dacoharkes@google.com>
This commit is contained in:
Vyacheslav Egorov 2022-07-06 12:48:16 +00:00 committed by Commit Bot
parent 1c7a7d0c41
commit a2ab7ccea4
2 changed files with 6 additions and 5 deletions

View file

@ -20,10 +20,9 @@ inline uint32_t FinalizeHash(uint32_t hash, intptr_t hashbits = kBitsPerInt32) {
hash += hash << 3;
hash ^= hash >> 11; // Logical shift, unsigned hash.
hash += hash << 15;
// FinalizeHash gets called with values for hashbits that are bigger than 31
// (like kBitsPerWord - 1). Therefore we are careful to use a type
// (uintptr_t) big enough to avoid undefined behavior with the left shift.
hash &= (static_cast<uintptr_t>(1) << hashbits) - 1;
if (hashbits < kBitsPerInt32) {
hash &= (static_cast<uint32_t>(1) << hashbits) - 1;
}
return (hash == 0) ? 1 : hash;
}

View file

@ -11,7 +11,9 @@ namespace dart {
class TestValue {
public:
explicit TestValue(intptr_t x) : x_(x) {}
uword Hash() const { return x_ & 1; }
// FinalizeHash is used here to provide coverage for FinalizeHash(...)
// function.
uword Hash() const { return FinalizeHash(static_cast<uint32_t>(x_) & 1); }
bool Equals(const TestValue& other) { return x_ == other.x_; }
private: