struct and memoryview tests rely on undefined behavior (as revealed by clang 9) #83870
Comments
|
The clang build was recently added for that buildbot and it seems on that particular architecture, test_struct fails with: ====================================================================== Traceback (most recent call last):
File "/home/dje/cpython-buildarea/3.x.edelsohn-fedora-rawhide-z.clang-ubsan/build/Lib/test/test_struct.py", line 520, in test_bool
self.assertTrue(struct.unpack('>?', c)[0])
AssertionError: False is not truehttps://buildbot.python.org/all/#/builders/488/builds/6 Fedora rawhide recently upgraded Clang to version 10. The rest of the architectures seem fine. |
|
Failed assertion here: https://github.com/python/cpython/blob/master/Lib/test/test_struct.py#L520 |
|
On this loop: for c in [b'\x01', b'\x7f', b'\xff', b'\x0f', b'\xf0']:
self.assertTrue(struct.unpack('>?', c)[0])It fails for the b'\xf0' case |
|
The call: static PyObject *
nu_bool(const char *p, const formatdef *f)
{
_Bool x;
memcpy((char *)&x, p, sizeof x);
return PyBool_FromLong(x != 0);
}i.e., copies "sizeof x" (1 byte) of memory to a temporary buffer x, and then treats that as _Bool. While I don't have access to the C standard, I believe it says that assignment of a true value to _Bool can coerce to a unique "true" value. It seems that if a char doesn't have the exact bit pattern for true or false, casting to _Bool is undefined behavior. Is that correct? Clang 10 on s390x seems to take advantage of this: it probably only looks at the last bit(s) so a _Bool with a bit pattern of 0xf0 turns out false. |
|
C compiler dev that it's indeed undefined behavior.
I'm left with a question for CPython's struct experts: The above would be my preferred fix, but the Python code is asking to convert a memory buffer to bool *using platform-specific semantics*. (Also, this assumes size of _Bool is the same as size of char. |
|
maybe we should be raising an error if the bytes are not a valid platform _Bool pattern? |
gpshead
added
extension-modules
|
the concept of a native _Bool seems fuzzy. the important thing for the struct module is to consume sizeof _Bool bytes from the input stream. how those are interpreted is up to the platform. So if the platform says a bool is 8 bytes and it only ever looks at the lowest bit in those for bool-ness, good for it. in that situation our unittest assuming that b'\xf0' should be true when interpreted as a bool is wrong. just get rid of that value from the loop in the test? |
|
IMO:
But what is "format"? The docs talk about size, alignment and byte order; bool representation is a slightly different concept. I'm not sure if it should follow Byte order or Size/Alignment: I think that the latter would be better (so only "@" uses the native _Bool semantics, but "=" uses portable char semantics), but it might be be harder to implement. |
|
Charalampos Stratakis also reported this issue to python-dev: |
|
fwiw oss-fuzz also finds this on struct (via ubsan) https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=20949 struct.unpack('?', ' ') |
|
Viewing the oss-fuzz bug requires login. Is there any interesting public info in it? |
|
Nothing too interesting, here's the stack trace: /src/cpython3/Modules/_struct.c:487:28: runtime error: load of value 32, which is not a valid value for type '_Bool' |
|
Using memcpy() to write a value different than 0 or 1 into a _Bool is clearly an undefined behavior. Example with clang UBSan. bool.c: #include <string.h>
#include <stdbool.h>
int main()
{
char ch = 42;
_Bool x;
memcpy(&x, &ch, 1);
return x == true;
}$ clang -fsanitize=bool bool.c -o bool
$ ./bool
bool.c:9:12: runtime error: load of value 42, which is not a valid value for type '_Bool' |
|
FYI - I updated the ubsan buildbot to clang 9 so this one shows up in there now: /var/lib/buildbot/workers/clang-ubsan/3.x.gps-clang-ubsan.clang-ubsan/build/Modules/_struct.c:487:28: runtime error: load of value 116, which is not a valid value for type '_Bool' |
That could be the proper thing to do, but it does make it easy to invoke C undefined behavior from Python code. AFAIK, it would be the only such place in the struct module. So, I'd like to assume and assert/test that sizeof(_Bool) is 1 and the false bit-pattern is (char)0, and with that, just use
That's quite hard to test for. |
|
On Wed, Mar 11, 2020, at 07:41, Petr Viktorin wrote:
How so? We just make the same assumption you're making that true = b'\x01' and false = NUL. |
|
The memoryview module does the same thing as struct, and its tests expect the same behavior as with struct. So, whatever we do in struct should be also done in memoryview. |
encukou
changed the title
Right, with that assumption, it's not that hard. And it becomes a test-only assumption, which is great! But, I'm still not convinced this would be a good fix. The current struct documentation is consistent with *casting* char to _Bool, rather than doing the memcpy and reinterpreting as _Bool. The memcpy makes sense for larget types, but not so much for _Bool. (Certainly, the docs' assertion that "the conversion between C and Python values should be obvious given their types" is not true here...) |
|
Oh! I just reead the docs more carefully; they say: For the '?' format character, the return value is either True or False. When packing, the truth value of the argument object is used. Either 0 or 1 in the native or standard bool representation will be packed, and any non-zero value will be True when unpacking. |
|
Concerning memoryview, I've looked at this very briefly: memoryview #define PACK_SINGLE(ptr, src, type) \
do { \
type x; \
x = (type)src; \
memcpy(ptr, (char *)&x, sizeof x); \
} while (0)This macro has exposed compiler bugs before. |
|
So which memoryview test (unless it is using the struct module) |
|
Okay, in memoryview the cast tests can trigger UB checks. memoryview assumes that bool is packed correctly, so just casting does not work. diff --git a/Lib/test/test_buffer.py b/Lib/test/test_buffer.py
index 6178ffde7a..86cf4c309f 100644
--- a/Lib/test/test_buffer.py
+++ b/Lib/test/test_buffer.py
@@ -2758,6 +2758,8 @@ class TestBufferProtocol(unittest.TestCase):
tsize = struct.calcsize(tfmt)
n = prod(_tshape) * tsize
obj = 'memoryview' if is_byte_format(tfmt) else 'bytefmt'
+ if "?" in tfmt:
+ continue
for fmt, items, _ in iter_format(n, obj):
size = struct.calcsize(fmt)
shape = [n] if n > 0 else [] |
|
I checked that NumPy also packs correctly: >>> import numpy as np
>>> x = np.array([0,1,2,3], dtype=np.bool)
>>> x.tobytes()
b'\x00\x01\x01\x01'So I vote for not handling incorrectly packed values and removing memoryview also does not guard against the theoretical possibility |
I disagree. I don't think struct module's job is to be faithful to _Bool semantics. Up to this point, "?" worked for bytes with "only 0 is false" semantics, in a reliable and documented way. I don't see a reason to let that continue. You're right about trap representations, but IMO floats are much more tied to hardware (and serious users of float are aware of the pitfalls), while _Bool semantics are governed by the whims of the compiler. Also, the "@" prefix is specifically documented to select native Byte order, Size, and Alignment; not how the bit-pattern is interpreted. |
|
The docs for native mode (we now always assume C99): "The '?' conversion code corresponds to the _Bool type defined by C99." The memoryview tests that fail are essentially auto-generated and not >>> x = array.array("B", [0,1,2,3])
>>> m = memoryview(x)
>>> m.format
'B'
>>> c = m.cast("?") # Wrong!
>>> c.tolist()
[False, True, True, True]I don't see the point in working around UB for the purposes of If you do subsequent array operations with the variable "c", UB will happen there. In theory you can do the same with a cast from unsigned to signed |
But, nowhere is it suggested that conversion is "do memcpy and then interpret the bit pattern". That is pretty weird way to convert values in C; it seems to me that it's only a hack to avoid unaligned access. |
|
The memcpy() is NOT a hack and performs exactly the same operation On platforms like x86 the memcpy() is optimized to a simple assignment. Casting the pointer and then dereferencing would also be subject You are the one who wanted to *introduce* a hack by dereferencing |
|
memoryview only supports the native format, so I've disabled the IMO the problem in _struct is that it swaps the x->unpack function If one disables that swap, the tests pass here. |
Yes, I did change my mind after reading the documentation. The docs say two contradicting things:
So it's clear that something has to change. IMO, preserving (2) and relaxing (1) is the more useful choice. |
But not in this issue I think. #63169 is a minimal change that UB for the native type is a direct consequence of using _Bool. Also, UB can only happen in a constructed example --- correctly So I think any fear of UB here is not warranted. |
|
Note that the implementation of np_bool in _struct.c [1] is incorrect because this is supposed to access a boolean of a standard size, but uses _Bool. The size of _Bool is not prescribed, and IIRC sizeof(_Bool) was 4 with the compilers used for macOS/PPC. [1] https://github.com/python/cpython/blob/master/Modules/_struct.c#L703 |
|
Sigh... never mind, I misread the code. Please ignore msg364472 |
|
I think we are speaking past each other. In my (current) view, the semantics are spelled out in the documentation: "any non-zero value will be True when unpacking". In your view, the semantics are dictated by the correspondence to _Bool, and the "non-zero value will be True when unpacking" is the fluff to be ignored and removed. The docs assume the two behaviors (_Bool and non-zero) are equivalent. In this bug we find out that they are not, so to fix the bug, we need to make a choice which one to keep and which one to throw out. What "array libraries expect" is IMO not relevant: under any of the two views, libraries that are well-written (under that view) will be fine. Problems come when the library and Python choose different sides, e.g. when a non-C library can't use _Bool and thus packs arrays with the expectation that "any non-zero value will be True when unpacking". What is a minimal change in *implementation* is a bigger change in *behavior*: unpacking of arrays will now depend greatly on details like the compiler used to build Python. I see that as the greater evil: since the data can be sharted across environments, languages and compilers, keeping the semantics well-defined seems better than leaving them to the compiler. |
|
I think this issue should be about fixing the tests so that people #63169 fixes "<?", ">?" and "!?", which clearly used wrong Can we please discuss native _Bool in another issue? There is no non-hackish way of unpacking _Bool if new compilers You could determine sizeof(_Bool), use the matching unsigned type, I'd rather deprecate _Bool and use uint8_t, but that definitely |
|
I see. Thanks for your patience explaining this to me! I will merge and continue in a different issue. |
|
Moved to Discourse, IMO that's a better place for maintainers of other PEP-3118-compatible libraries to chime in: |
|
Objects/memoryview.c uses memcpy() on _Bool which leads to undefined behavior with GCC 11: see bpo-42587. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: