Xtrabackup tests are failing

[deepthi]

Overview of the Issue

Cluster(20) has been failing for the last 12 hours or so, with most of the xtrabackup tests failing. The ones using xbstream are still passing.
Since we download the latest binary in CI, this may have nothing to do with our code, but with some change to the binary. Needs to be investigated.

Reproduction Steps

See https://github.com/vitessio/vitess/actions/workflows/cluster_endtoend_20.yml

Binary Version

main

Operating System and Environment details

CI

Log Fragments

I0426 21:08:30.081012   18259 vtctlclient_process.go:209] Executing vtctlclient with command: vtctlclient --server localhost:16006 Backup zone1-0000002642 (attempt 1 of 10)
E0426 21:08:33.658627   18259 vtctlclient_process.go:186] Output:
E0426 21:08:33.657708   21427 main.go:94] E0426 21:08:33.657563 backup.go:107] E0426 21:08:33.657289 backup.go:127] backup is not usable, aborting it: [exit status 2
xtrabackup failed with error. Output=]
Backup Error: rpc error: code = Unknown desc = TabletManager.Backup on zone1-0000002642 error: xtrabackup failed with error. Output=: exit status 2: xtrabackup failed with error. Output=: exit status 2
E0426 21:08:33.657991   21427 main.go:103] remote error: rpc error: code = Unknown desc = TabletManager.Backup on zone1-0000002642 error: xtrabackup failed with error. Output=: exit status 2: xtrabackup failed with error. Output=: exit status 2
    backup_utils.go:603: 
        	Error Trace:	backup_utils.go:603
        	            				backup_utils.go:222
        	Error:      	Expected nil, but got: &exec.ExitError{ProcessState:(*os.ProcessState)(0xc0005660a8), Stderr:[]uint8(nil)}
        	Test:       	TestXtrabackup/TestReplicaBackup

[deepthi]

@mattlord pointed out that a new release came out today
https://docs.percona.com/percona-xtrabackup/2.4/release-notes/2.4/2.4.25.html

[mattlord]

Here's the problem. xtrabackup is crashing:

xtrabackupengine.go:147] Detected MySQL flavor: MySQL56
xtrabackupengine.go:157] Starting backup with 0 stripe(s)
xtrabackupengine.go:311] xtrabackup stderr: xtrabackup: recognized server arguments: --datadir=/vt/vtdataroot/vtroot_7301/vt_0000006411/data --innodb_data_home_dir=/vt/vtdataroot/vtroot_7301/vt_0000006411/innodb/data --innodb_log_group_home_dir=/vt/vtdataroot/vtroot_7301/vt_0000006411/innodb/logs --log_bin=/vt/vtdataroot/vtroot_7301/vt_0000006411/bin-logs/vt-0000006411-bin --server-id=1885372146 --tmpdir=/vt/vtdataroot/vtroot_7301/vt_0000006411/tmp
xtrabackupengine.go:311] xtrabackup stderr: xtrabackup: recognized client arguments: --backup=1 --socket=/vt/vtdataroot/vtroot_7301/vt_0000006411/mysql.sock --slave-info=1 --user=vt_dba --target-dir=/vt/vtdataroot/vtroot_7301/vt_0000006411/tmp --stream=tar --password=*
xtrabackupengine.go:311] xtrabackup stderr: 220427 03:19:06  version_check Connecting to MySQL server with DSN 'dbi:mysql:;mysql_read_default_group=xtrabackup;mysql_socket=/vt/vtdataroot/vtroot_7301/vt_0000006411/mysql.sock' as 'vt_dba'  (using password: YES).
xtrabackupengine.go:311] xtrabackup stderr: 220427 03:19:06  version_check Connected to MySQL server
xtrabackupengine.go:311] xtrabackup stderr: 220427 03:19:06  version_check Executing a version check against the server...
xtrabackupengine.go:311] xtrabackup stderr: 220427 03:19:06  version_check Done.
xtrabackupengine.go:311] xtrabackup stderr: 220427 03:19:06 Connecting to MySQL server host: localhost, user: vt_dba, password: set, port: not set, socket: /vt/vtdataroot/vtroot_7301/vt_0000006411/mysql.sock
xtrabackupengine.go:311] xtrabackup stderr: Using server version 5.7.37-log
xtrabackupengine.go:311] xtrabackup stderr: xtrabackup version 2.4.25 based on MySQL server 5.7.35 Linux (x86_64) (revision id: 90fe9d0)
xtrabackupengine.go:311] xtrabackup stderr: xtrabackup: uses posix_fadvise().
xtrabackupengine.go:311] xtrabackup stderr: xtrabackup: cd to /vt/vtdataroot/vtroot_7301/vt_0000006411/data
xtrabackupengine.go:311] xtrabackup stderr: xtrabackup: open files limit requested 0, set to 65535
xtrabackupengine.go:311] xtrabackup stderr: xtrabackup: using the following InnoDB configuration:
xtrabackupengine.go:311] xtrabackup stderr: xtrabackup:   innodb_data_home_dir = /vt/vtdataroot/vtroot_7301/vt_0000006411/innodb/data
xtrabackupengine.go:311] xtrabackup stderr: xtrabackup:   innodb_data_file_path = ibdata1:12M:autoextend
xtrabackupengine.go:311] xtrabackup stderr: xtrabackup:   innodb_log_group_home_dir = /vt/vtdataroot/vtroot_7301/vt_0000006411/innodb/logs
xtrabackupengine.go:311] xtrabackup stderr: xtrabackup:   innodb_log_files_in_group = 2
xtrabackupengine.go:311] xtrabackup stderr: xtrabackup:   innodb_log_file_size = 50331648
xtrabackupengine.go:311] xtrabackup stderr: InnoDB: Number of pools: 1
xtrabackupengine.go:311] xtrabackup stderr: 03:19:06 UTC - xtrabackup got signal 11 ;
xtrabackupengine.go:311] xtrabackup stderr: This could be because you hit a bug or data is corrupted.
xtrabackupengine.go:311] xtrabackup stderr: This error can also be caused by malfunctioning hardware.
xtrabackupengine.go:311] xtrabackup stderr: Attempting to collect some information that could help diagnose the problem.
xtrabackupengine.go:311] xtrabackup stderr: As this is a crash and something is definitely wrong, the information
xtrabackupengine.go:311] xtrabackup stderr: collection process might fail.
xtrabackupengine.go:311] xtrabackup stderr:
xtrabackupengine.go:311] xtrabackup stderr: Thread pointer: 0x0
xtrabackupengine.go:311] xtrabackup stderr: Attempting backtrace. You can use the following information to find out
xtrabackupengine.go:311] xtrabackup stderr: where mysqld died. If you see no messages after this, something went
xtrabackupengine.go:311] xtrabackup stderr: terribly wrong...
xtrabackupengine.go:311] xtrabackup stderr: stack_bottom = 0 thread_stack 0x10000
xtrabackupengine.go:311] xtrabackup stderr: xtrabackup(my_print_stacktrace+0x2c)[0xd730ec]
xtrabackupengine.go:311] xtrabackup stderr: xtrabackup(handle_fatal_signal+0x286)[0xa37116]
xtrabackupengine.go:311] xtrabackup stderr: /lib64/libpthread.so.0(+0x117e0)[0x7f46d12ef7e0]
xtrabackupengine.go:311] xtrabackup stderr: xtrabackup(my_tmpdir+0x4e)[0xd69cae]
xtrabackupengine.go:311] xtrabackup stderr: xtrabackup[0x738b94]
xtrabackupengine.go:311] xtrabackup stderr: xtrabackup(ds_open+0x12)[0x735b42]
xtrabackupengine.go:311] xtrabackup stderr: xtrabackup(_Z22xtrabackup_backup_funcv+0xda0)[0x72aa60]
xtrabackupengine.go:311] xtrabackup stderr: xtrabackup(main+0xa94)[0x704c44]
xtrabackupengine.go:311] xtrabackup stderr: /lib64/libc.so.6(__libc_start_main+0xea)[0x7f46cf0d713a]
xtrabackupengine.go:311] xtrabackup stderr: xtrabackup[0x71ee04]
xtrabackupengine.go:311] xtrabackup stderr:
xtrabackupengine.go:311] xtrabackup stderr: Please report a bug at https://jira.percona.com/projects/PXB
xtrabackupengine.go:117] Closing backup file backup.tar.gz

rpc_server.go:80] TabletManager.Backup(concurrency:4)(on zone1-0000006411 from ) error: xtrabackup failed with error. Output=: exit status 2

[mattlord]

We seem to be getting a SEGFAULT / SIGSEGV here: https://github.com/percona/percona-xtrabackup/blob/2.4/storage/innobase/xtrabackup/src/ds_tmpfile.c#L94-L111

It somehow seems to have been caused by this PR that is only in the brand new release: percona/percona-xtrabackup@eb29675

[mattlord]

I filed a bug upstream: https://jira.percona.com/browse/PXB-2756

For now I think we will have to try and pin the xtrabackup version we use in the CI to 2.4.24. I will look into that tomorrow.

[mattlord]

PR for the fix is here: percona/percona-xtrabackup#1314