You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This is strange: runs ok in CI but recently started failing on my local. Fails for 32 bits, max of 1.0. Looks like issues with floating point error but idk what is different now, needs investigating.
The text was updated successfully, but these errors were encountered:
Not sure why the issue only appears sometimes but I think the current solution is just not correct; the correct types and order of operation to get the expected values at 0 and MAX are:
LinearQuantizer::LinearQuantizer(float max, std::uint8_t bits)
: m_range((1UL << bits) - 1U), m_max(max) {
if (max <= 0.0) {
throw std::runtime_error(
fmt::format("Max score for linear quantizer must be positive but {} passed", max)
);
}
if (bits > 32 or bits < 2) {
throw std::runtime_error(fmt::format(
"Linear quantizer must take a number of bits between 2 and 32 but {} passed", bits
));
}
}
auto LinearQuantizer::operator()(float value) const -> std::uint32_t {
if (value < 0 || value > m_max) {
throw std::invalid_argument(
fmt::format("quantized value must be between 0 and {} but {} given", m_max, value)
);
}
// This is always in (0, 1] range.
auto normalized_value = static_cast<double>(value / m_max);
return static_cast<std::uint32_t>(normalized_value * (m_range - 1)) + 1;
}
This removes scale which may introduce some errors imo, and casts the normalized value to double, so that once it's multiplied by range, it doesn't lose precision.
This is strange: runs ok in CI but recently started failing on my local. Fails for 32 bits, max of 1.0. Looks like issues with floating point error but idk what is different now, needs investigating.
The text was updated successfully, but these errors were encountered: