Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix writer precision when fp bits >= 14 #909

Merged
merged 2 commits into from
Nov 27, 2023

Conversation

calad0i
Copy link
Contributor

@calad0i calad0i commented Nov 3, 2023

A# Description

In the current version, #decimals to write is determined by

lsb = 2**-new_precision.fractional
decimal_spaces = len(str(lsb).split('.')[1])

However, str(2**-14)='6.103515625e-05', which has 13 instead of 14 digits, leads to bit inaccuracy in some cases.

Indeed, #decimal points required is exactly #fp bits (easy to check: x., x.5, x.x5, x.xx5, ...)

Type of change

  • Bug fix (non-breaking change that fixes an issue)

Tests

test/pytest/test_weight_writer.py

Checklist

  • I have read the guidelines for contributing.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have made corresponding changes to the documentation.
  • My changes generate no new warnings.
  • I have installed and run pre-commit on the files I edited or added.
  • I have added tests that prove my fix is effective or that my feature works.

@calad0i calad0i mentioned this pull request Nov 7, 2023
7 tasks
@vloncar vloncar added this to the v0.8.1 milestone Nov 10, 2023
@jmitrevs jmitrevs added the please test Trigger testing by creating local PR branch label Nov 27, 2023
@jmitrevs jmitrevs merged commit 22a1054 into fastmachinelearning:main Nov 27, 2023
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
please test Trigger testing by creating local PR branch
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants