system/serenity
system/serenity
1
0
Fork 0
mirror of https://github.com/SerenityOS/serenity synced 2024-10-02 22:24:26 +00:00
Code Issues Packages Projects Releases Wiki Activity

AK: Fix off-by-one error in round-to-even logic of FixedPoint

Browse source 
Because of the off-by-one error, the second bit of the fraction was
getting ignored in differentiating between fractions equal to 0.5 or
greater than 0.5. This resulted in numbers like 2.75 being considered
as having fraction equal to 0.5 and getting rounded incorrectly (to 2).
This commit is contained in:
ronak69 2023-08-12 15:50:52 +00:00 committed by Andrew Kaster
parent 6ef2c34eb4
commit 4c6ea4a963
2 changed files with 5 additions and 2 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

4
AK/FixedPoint.h
View file

@ -111,7 +111,7 @@ public:
// fract(m_value) >= .5?
if (m_value & (1u << (precision - 1))) {
// fract(m_value) > .5?
if (m_value & (radix_mask >> 2u)) {
if (m_value & (radix_mask >> 1)) {
// yes: round "up";
value += 1;
} else {
@ -219,7 +219,7 @@ public:
// If last bit cut off is 1:
if (value & (1u << (precision - 1))) {
// If the bit after is 1 as well
if (value & (radix_mask >> 2u)) {
if (value & (radix_mask >> 1)) {
// We round away from 0
ret.raw() += 1;
} else {

3
Tests/AK/TestFixedPoint.cpp
View file

@ -58,6 +58,9 @@ TEST_CASE(rounding)
EXPECT_EQ(Type(-1.5).ceil(), Type(-1));
EXPECT_EQ(Type(-1.25).trunc(), Type(-1));
EXPECT_EQ(Type(2.75).rint(), Type(3));
EXPECT_EQ(Type(-1.25).rint(), Type(-1));
EXPECT_EQ(Type(0.5).lrint(), 0);
EXPECT_EQ(Type(0.5).lfloor(), 0);
EXPECT_EQ(Type(0.5).lceil(), 1);