Segmentation fault caused by updating to 8.1

[JimmyBanks]

Description

All PHP tests on local using Unit testing etc. function correctly.

However, once put into production our server started throwing thousands of the following segfault errors, and only returns null responses from our MySQL database connection, resulting in approximately 50% of our requests failing.

php8.1-fpm.log is throwing thousands of these errors:
WARNING: [pool www] child 2971 exited on signal 11 (SIGSEGV - core dumped) after 469.405694 seconds from start

Similarly nginx's error log is throwing thousands of:

"GET /forums/thread/5458 HTTP/2.0", upstream: "fastcgi://unix:/run/php/php8.1-fpm.sock:", host: "www.example.com", referrer: "https://www.example.com/machine/10"

Neither of these errors are occurring on PHP8.0. We had to downgrade for obvious reasons.

PHP Version

8.1.1

Operating System

Ubuntu 18.04

[damianwadley]

Can you generate and post a backtrace of the crash?

[JimmyBanks]

@damianwadley

As soon as I had those logs recorded I reverted to 8.0 as we were losing so many requests. So I've lost the VMs that had the issue. I'll push an 8.1 VM to staging and see if I can get the error to occur without production traffic.

[JimmyBanks]

@damianwadley

After spending all of today researching, I am wondering if this could be OPCache related, do you know if 8.1 uses more OPCache memory space than 8.0?

Looking at my settings with php-i, currently opcache.memory_consumption is only set to 128M

I also utilize preloading.

[damianwadley]

Hard to say. To start investigating down that path, you could disable opcache and see what happens.

[JimmyBanks]

@damianwadley

I have gotten the backtraces, there seems to be multiple different backtraces being generated by the segfaults, I have found three unique backtrace patterns, with the three line one consisting of 90% of the crashes.

This is all new to me, I've never run into a segfault issue before, but for whatever reason the backtrace only has double question marks. So is this useful, or do I need to do something else to get further details for the backtrace to be useful?

[damianwadley]

You're right that the question marks don't make it very useful. When you started gdb with the core dump, did you see messages about it trying to load symbols from files, some of which didn't exist? If so then that's probably what's getting in the way, so if you can, use gdb on the machine that generated the core dump (and thus has the files).

[JimmyBanks]

@damianwadley

btCore was generated by `php-fpm: pool www                                                            '.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f72134fa743 in Arena_Get (val=val@entry=0x7f7226552c18)
    at /tmp/pear/temp/protobuf/arena.c:72
72      /tmp/pear/temp/protobuf/arena.c: No such file or directory.
[Current thread is 1 (Thread 0x7f722c618500 (LWP 1938))]
(gdb) bt
#0  0x00007f72134fa743 in Arena_Get (val=val@entry=0x7f7226552c18)
    at /tmp/pear/temp/protobuf/arena.c:72
#1  0x00007f7213506064 in zim_Message___construct (
    execute_data=0x7f72264148c0, return_value=<optimized out>)
    at /tmp/pear/temp/protobuf/message.c:579
#2  0x000055ce3ea58db2 in execute_ex ()
#3  0x000055ce3ea5920d in zend_execute ()
#4  0x000055ce3e9e69c4 in zend_execute_scripts ()
#5  0x000055ce3e982791 in php_execute_script ()
#6  0x000055ce3e81dbbd in ?? ()
#7  0x00007f72299a9bf7 in __libc_start_main (main=0x55ce3e81cda0, argc=4,
    argv=0x7ffe9219a1e8, init=<optimized out>, fini=<optimized out>,
    rtld_fini=<optimized out>, stack_end=0x7ffe9219a1d8)
    at ../csu/libc-start.c:310
#8  0x000055ce3e81e9ea in _start ()

Core was generated by `php-fpm: pool www                                                            '.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  _upb_arenahas (a=<optimized out>) at /tmp/pear/temp/protobuf/php-upb.h:448
448     /tmp/pear/temp/protobuf/php-upb.h: No such file or directory.
[Current thread is 1 (Thread 0x7f722c618500 (LWP 1959))]
(gdb) bt
#0  _upb_arenahas (a=<optimized out>) at /tmp/pear/temp/protobuf/php-upb.h:448
#1  upb_arena_malloc (size=<optimized out>, a=<optimized out>)
    at /tmp/pear/temp/protobuf/php-upb.h:456
#2  _upb_msg_new_inl (a=0x78, l=0x55ce40ce37b0)
    at /tmp/pear/temp/protobuf/php-upb.h:1211
#3  _upb_msg_new (l=0x55ce40ce37b0, a=0x78)
    at /tmp/pear/temp/protobuf/php-upb.c:1493
#4  0x00007f7213513435 in upb_msg_new (m=<optimized out>, a=<optimized out>)
    at /tmp/pear/temp/protobuf/php-upb.c:7047
#5  0x00007f72134ff9c5 in Message_Initialize (
    intern=intern@entry=0x7f7226402fc8, desc=desc@entry=0x7f72264728c0)
    at /tmp/pear/temp/protobuf/message.c:565
#6  0x00007f72135060a3 in zim_Message___construct (
    execute_data=0x7f7226413e40, return_value=<optimized out>)
    at /tmp/pear/temp/protobuf/message.c:602
#7  0x000055ce3ea58db2 in execute_ex ()
#8  0x000055ce3ea5920d in zend_execute ()
#9  0x000055ce3e9e69c4 in zend_execute_scripts ()
#10 0x000055ce3e982791 in php_execute_script ()
#11 0x000055ce3e81dbbd in ?? ()
#12 0x00007f72299a9bf7 in __libc_start_main (main=0x55ce3e81cda0, argc=4,
    argv=0x7ffe9219a1e8, init=<optimized out>, fini=<optimized out>,
    rtld_fini=<optimized out>, stack_end=0x7ffe9219a1d8)

Appears to be related to the pear protobuf package, which I have installed with pecl:

sudo pecl install protobuf
echo "extension=protobuf.so" | sudo tee -a /etc/php/8.1/fpm/conf.d/protobuf.ini

Version is 3.19.3

[bwoebi]

In that case I'd recommend reporting the bug against https://github.com/protocolbuffers/protobuf/issues directly, we cannot do much here.

[JimmyBanks]

For anybody following along bug report has been issued here: protocolbuffers/protobuf#9446

[damianwadley]

Indeed, it does look to be pretty squarely an issue within the protobuf extension.