Perl crashes when used with DBIx::Connector

[bit0mike]

This is the same issue as ap/DBIx-Connector#50 as it's not clear which module is causing the problem.

Trying to use DBD::MariaDB in place of DBD::mysql results in some weird behavior sometimes, notably Perl crashing (SIGBUS core dump, on FreeBSD at least) when disconnecting from the database when mixed with Parallel::ForkManager. Obviously normally Parallel::ForkManager shouldn't be used with open DBI handles, but that's one of the things DBIx::Connector is supposed to make "safe". And with the DBD::mysql driver, it is.

DBIx::Connector in its release version does not have a specific driver for MariaDB. In the master branch, there is a basic one that is more or less a duplicate of the mysql one. Using it doesn't fix the problem.

Script to reproduce:

#!/usr/local/bin/perl
use v5.38;
use Parallel::ForkManager;
use File::Temp;
use DBIx::Connector;
my $user = 'REDACTED';
my $pass = 'REDACTED';
my $forkmgr = Parallel::ForkManager->new( 32, File::Temp->newdir() );
my $dbix_obj = DBIx::Connector->new(
"DBI:MariaDB:database=test;host=localhost;mariadb_connect_timeout=10", $user, $pass, # dumps core sometimes
#"DBI:mysql:database=test;host=localhost;mysql_connect_timeout=10", $user, $pass, # does not dump core ever
{ AutoCommit => 1, RaiseError => 1, ShowErrorStatement => 1, Callbacks => {} }
);
$dbix_obj->mode('fixup');
$dbix_obj->dbh; # Comment out to stop core dumps with DBD::MariaDB
foreach my $c ( 0 .. 5000 ) {
unless ( $forkmgr->start ) {
my ($q) = $dbix_obj->run( fixup => sub { $_->selectall_array( 'SELECT MD5(?)', undef, $c ) } );
say "pid=$$ c=$c result=$$q[0]";
$forkmgr->finish(0);
}
}
$forkmgr->wait_all_children;

On FreeBSD 14.1 this code will create MANY coredumps, all apparently when the child process DB handles try to disconnect. The code does run successfully anyway (but obviously slowly because of all the core files being written) because the crash is when the child is trying to exit anyway.

On Ubuntu 24.04 this code instead prints this at every DB disconnect -- but not every time the program runs, just sometimes:

panic: DBI active kids (-1) < 0 or > kids (0) at /usr/lib/x86_64-linux-gnu/perl5/5.38/DBI.pm line 759.

To stop that warning (Ubuntu) or coredumps (FreeBSD) any one of the following works:

use DBD::mysql (version 4.x; 5.x doesn't work w/ MariaDB anymore) instead of DBD::MariaDB. I don't know why there'd be a difference between the two, but, there is. This is why I went down the path of adding DBIx::Connector::Driver::MariaDB in the first place, to see if it would help. It did not, but it's good to have anyway.

OR

move DBIx::Connector->new inside the $forkmgr->start block, which I know is more "correct" to do (forking with open handles of ANY kind is bad, after all), but making it fork-safe outside the block is one of the whole points of using DBIx::Connector in the first place

OR

remove the $dbix_obj->dbh call immediately after connecting. I was doing this to make sure it could really connect, vs trying to do it lazily later on -- if the DB is really truly down, I want to know up front instead of later on. There could be a better way of accomplishing this test though (ping?), if this is a seriously stupid thing for me to be doing this way, I'm open to suggestions

[Grinnz]

DBIx::Connector does not inherently make database handles fork-safe; they should never be reused across processes, the only safety feature DBI provides for this is AutoInactiveDestroy, which should generally always be used and DBIx::Connector sets by default. This prevents child processes from destroying a database handle opened by the parent. DBIx::Connector additionally stores the PID where the database handle was opened and avoids reusing it in a different process.

Since this is occurring when retrieving a dbh that then gets cached in the DBIx::Connector object, and that dbh will never be used by DBIx::Connector in the child, it seems to me that the issue is related to InactiveDestroy in the child.

Attempting to retrieve a dbh is a perfectly reasonable way of determining connectivity to the database upfront. As a suggestion to work around this bug, I would suggest creating a secondary DBIx::Connector object for this purpose, allow it to fall out of scope before child processes are forked (such as undefing the variable), and then recreate another DBIx::Connector object either in the parent or the child processes for use in the forks (up to preference).

[ap]

@Grinnz

it seems to me that the issue is related to InactiveDestroy in the child. […] As a suggestion to work around this bug

I’m not deep enough in this subject matter right now to fully follow – are you saying that this is a bug in DBD::MariaDB rather than DBIx::Connector?

[Grinnz]

That is my best guess - otherwise it is some interaction between DBD::MariaDB's destroy code and how DBIx::Connector works, but I doubt DBIx::Connector is doing anything that interesting there.

[ap]

Thanks.

[choroba]

If it is InactiveDestroy, the fix in https://github.com/perl5-dbi/DBD-MariaDB/compare/master...choroba:inactive-destroy?expand=1 might help. Can anyone test it?

[bit0mike]

Ok, testing with the master versions of DBD::MariaDB and DBIx::Connector, I'm not seeing any difference -- still core dumps on FreeBSD, still the same error (and an occasional not-noticed-until-now coredump) on Ubuntu.

The coredump on FreeBSD looks like this:

 (lldb) target create "/usr/local/bin/perl" --core "/var/tmp/perl.17317.1007.core"
 Core file '/var/tmp/perl.17317.1007.core' (x86_64) was loaded.
 (lldb) bt all
 * thread #1, name = 'perl', stop reason = signal SIGBUS
   * frame #0: 0x0000000800b1ee68 MariaDB.so`mariadb_db_close_mysql + 24
     frame #1: 0x0000000800b1f17b MariaDB.so`mariadb_dr_discon_all + 139
     frame #2: 0x0000000800b18435 MariaDB.so`XS_DBD__MariaDB__dr_discon_all_ + 117
     frame #3: 0x0000000800af8463 DBI.so`XS_DBI_dispatch + 8083
     frame #4: 0x00000008004f9550 libperl.so.5.40`___lldb_unnamed_symbol3485 + 2560
     frame #5: 0x0000000800575fd3 libperl.so.5.40`Perl_runops_standard + 67
     frame #6: 0x000000080046a0ce libperl.so.5.40`Perl_call_sv + 958
     frame #7: 0x0000000800466113 libperl.so.5.40`Perl_call_list + 467
     frame #8: 0x00000008004644ff libperl.so.5.40`perl_destruct + 367
     frame #9: 0x0000000000201da7 perl`main + 183
     frame #10: 0x00000008006d4a6a libc.so.7`__libc_start1 + 298
     frame #11: 0x0000000000201c80 perl`_start + 48

If anyone wants a run on either OS with DBI->trace(4) or something similar, let me know.

[Grinnz]

The fix @choroba mentioned is on the choroba:inactive-destroy branch, not master, currently.

[bit0mike]

Thank you for calling me out on my inability to read. Derp! ;-)

Good news: that version of dbdimp.c does seem to fix everything. \o/