Fix writer precision when fp bits >= 14 #909
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
A# Description
In the current version, #decimals to write is determined by
However, str(2**-14)='6.103515625e-05', which has 13 instead of 14 digits, leads to bit inaccuracy in some cases.
Indeed, #decimal points required is exactly #fp bits (easy to check: x., x.5, x.x5, x.xx5, ...)
Type of change
Tests
test/pytest/test_weight_writer.py
Checklist
pre-commit
on the files I edited or added.