libc++abi.dylib: Pure virtual function called!

[toktarev]

Note: Please use Issues only for bug reports. For questions, discussions, feature requests, etc. post to dev group: https://www.facebook.com/groups/rocksdb.dev

Expected behavior

Test finished successfully

Actual behavior

libc++abi.dylib: Pure virtual function called!

Steps to reproduce the behavior

#include "rocksdb/db.h"

`int main() {
system("rm -rf /repos/rocksdata/*");
system("exec rm -rf /repos/load.sst");

rocksdb::DB *dbh;

rocksdb::Options options{};

options.create_if_missing = true;

rocksdb::Status ostat = rocksdb::DB::Open(options, "/repos/rocksdata", &dbh);

rocksdb::ColumnFamilyHandle *value_handle;

dbh->CreateColumnFamily(rocksdb::ColumnFamilyOptions(), "test.value", &value_handle);

rocksdb::Options sstOptions{};

rocksdb::SstFileWriter *sst_file_writer = new rocksdb::SstFileWriter(rocksdb::EnvOptions(),
                                                                     sstOptions,
                                                                     sstOptions.comparator,
                                                                     value_handle);

rocksdb::Status s = sst_file_writer->Open("/repos/load.sst");

s = sst_file_writer->Put("1","1");

s = sst_file_writer->Finish();

rocksdb::IngestExternalFileOptions ingestOptions{};

s = dbh->IngestExternalFile(value_handle, {"/repos/load.sst"}, ingestOptions);

dbh->DropColumnFamily(value_handle);

dbh->DestroyColumnFamilyHandle(value_handle);

delete dbh;

}`

[ajkr]

couldn't repro.. can you run under gdb and get a backtrace?

[toktarev]

• thread Miss Spelling in README #1: tid = 0x17f7cd, 0x00007fff847f2286 libsystem_kernel.dylib`__pthread_kill + 10, queue = 'com.apple.main-thread', stop reason = signal SIGABRT
  • frame #0: 0x00007fff847f2286 libsystem_kernel.dylib__pthread_kill + 10 frame #1: 0x00007fff814c59f9 libsystem_pthread.dylibpthread_kill + 90
    frame OSX compilation #2: 0x00007fff85a659ab libsystem_c.dylibabort + 129 frame #3: 0x00007fff8eef5a21 libc++abi.dylibabort_message + 257
    frame compile error on Ubuntu #4: 0x00007fff8ef1c42f libc++abi.dylib__cxa_pure_virtual + 18 frame #5: 0x0000000100be7b6b libStarrocks.dylibrocksdb::ShardedCache::Erase(rocksdb::Slice const&) + 75
    frame make check fails #6: 0x0000000100defe0b libStarrocks.dylibrocksdb::BlockBasedTable::Close() + 379 frame #7: 0x0000000100de4399 libStarrocks.dylibrocksdb::BlockBasedTable::~BlockBasedTable() + 25
    frame Z_BUF_ERROR is not valid for BZip2_Compress #8: 0x0000000100de44ee libStarrocks.dylibrocksdb::BlockBasedTable::~BlockBasedTable() + 14 frame #9: 0x0000000100be54e3 libStarrocks.dylibrocksdb::LRUCacheShard::Erase(rocksdb::Slice const&, unsigned int) + 387
    frame EnvPosixTest.RandomAscessUniqueID test fails #10: 0x0000000100d01849 libStarrocks.dylibrocksdb::TableCache::Evict(rocksdb::Cache*, unsigned long long) + 41 frame #11: 0x0000000100c7bea6 libStarrocks.dylibrocksdb::DBImpl::PurgeObsoleteFiles(rocksdb::JobContext const&, bool) + 2854
    frame OSX Build failing = ./include/rocksdb/statistics.h:9:10: fatal error: 'atomic' file not found #12: 0x0000000100bed562 libStarrocks.dylibrocksdb::ColumnFamilyHandleImpl::~ColumnFamilyHandleImpl() + 562 frame #13: 0x0000000100bed5fe libStarrocks.dylibrocksdb::ColumnFamilyHandleImpl::~ColumnFamilyHandleImpl() + 14
    frame Impl Documentation missing #14: 0x0000000100c42f77 libStarrocks.dylibrocksdb::DB::DestroyColumnFamilyHandle(rocksdb::ColumnFamilyHandle*) + 23 frame #15: 0x00000001008a14de virtualmain + 2190 at virtual.cc:23
    frame Added missing component : gflags in Linux platform. #16: 0x00007fff875525c9 libdyld.dylib`start + 1

[toktarev]

RocksDB version: 5.8.7

[maysamyabandeh]

Seems like shared_ptr to block cache is mishandled here; the block cache was deleted before being accessed again here:
Delete stack trace:

  6 #0  rocksdb::LRUCache::~LRUCache (this=0x7ffff64dc790, __in_chrg=<optimized out>) at cache/lru_cache.cc:476
 15 #5  rocksdb::BlockBasedTableOptions::~BlockBasedTableOptions (this=0x7ffff66498c8, __in_chrg=<optimized out>) at ./include/rocksdb/table.h:52
 16 #6  rocksdb::BlockBasedTableFactory::~BlockBasedTableFactory (this=0x7ffff66498c0, __in_chrg=<optimized out>) at ./table/block_based_table_factory.h:34
 17 #7  rocksdb::BlockBasedTableFactory::~BlockBasedTableFactory (this=0x7ffff66498c0, __in_chrg=<optimized out>) at ./table/block_based_table_factory.h:34
 28 #13 rocksdb::ColumnFamilyOptions::~ColumnFamilyOptions (this=0x7ffff64ddc88, __in_chrg=<optimized out>) at ./include/rocksdb/options.h:81
 29 #14 rocksdb::ColumnFamilyData::~ColumnFamilyData (this=0x7ffff64ddc00, __in_chrg=<optimized out>) at db/column_family.cc:424
 30 #15 0x000000000053a5e8 in rocksdb::ColumnFamilyHandleImpl::~ColumnFamilyHandleImpl (this=0x7ffff65335a0, __in_chrg=<optimized out>) at db/column_family.cc:59

and accessed again at

#6  0x0000000000520121 in rocksdb::ShardedCache::Erase (this=0x7ffff64dc790, key=...) at cache/sharded_cache.cc:77
#7  0x00000000006a9d57 in rocksdb::BlockBasedTable::Close (this=0x7ffff6684b30) at table/block_based_table_reader.cc:2268
#8  0x00000000006a9db6 in rocksdb::BlockBasedTable::~BlockBasedTable (this=0x7ffff6684b30, __in_chrg=<optimized out>) at table/block_based_table_reader.cc:62
#9  0x00000000006a9fe1 in rocksdb::BlockBasedTable::~BlockBasedTable (this=0x7ffff6684b30, __in_chrg=<optimized out>) at table/block_based_table_reader.cc:64
#10 0x000000000051e55a in rocksdb::LRUHandle::Free (this=0x7ffff64a1700) at ./cache/lru_cache.h:108
#11 rocksdb::LRUCacheShard::Erase (this=0x7ffff64d5e40, key=..., hash=1004390799) at cache/lru_cache.cc:438
#12 0x0000000000600109 in rocksdb::TableCache::Evict (cache=<optimized out>, file_number=49) at db/table_cache.cc:469
#13 0x0000000000597513 in rocksdb::DBImpl::PurgeObsoleteFiles (this=0x7ffff64a9000, state=..., schedule_only=schedule_only@entry=false) at db/db_impl_files.cc:471
#14 0x000000000053a6a3 in rocksdb::ColumnFamilyHandleImpl::~ColumnFamilyHandleImpl (this=<optimized out>, __in_chrg=<optimized out>) at db/column_family.cc:64

[maysamyabandeh]

The problem seems to be that table reader keeps a reference of the options instead of a copy of them. As a result the shared_ptr to block cache is not ref counted and the pointer to block cache could be deleted while the table reader is still alive.

rocksdb/table/block_based_table_reader.h

Line 423 in 17e0403

const BlockBasedTableOptions& table_options;

[maysamyabandeh]

This is already fixed in master: 93d5269

[toktarev]

What release of RocksDB contains this fix ?

[toktarev]

I don't see this commit in latest version and before
https://github.com/facebook/rocksdb/commits/v5.12.4

We can't upgrage RocksDB version because of this bug