Missing Table for Tablet on 2.2->2.3 Upgrade

[nspiegelberg]

A user is facing an issue in Upgrade process, trying to migrate yugabyte from yugabyte-2.2.2.0 to yugabyte-2.3.2.0 on an existing data-directory.

./bin/yb-ctl start --data_dir=/path-to-datadir/yugabyte_datadir --timeout-processes-running-sec 1000 --timeout-yb-admin-sec 300
Starting cluster with base directory /path-to-datadir/yugabyte_datadir
Waiting for cluster to be ready.
Viewing file /path-to-datadir/yugabyte_datadir/node-2/disk-1/master.err:
F1016 09:19:44.019273 9239 catalog_manager.cc:695] Failed to load sys catalog: Corruption (yb/master/catalog_loaders.cc:157): Failed while visiting tablets in sys catalog: Missing table for tablet: : 04c4de3e6b5f47f3990f0a5f1218fe37
Fatal failure details written to /path-to-datadir/yugabyte_datadir/node-2/disk-1/yb-data/master/logs/yb-master.FATAL.details.2020-10-16T09_19_44.pid8982.txt
F20201016 09:19:44 ../../src/yb/master/catalog_manager.cc:695] Failed to load sys catalog: Corruption (yb/master/catalog_loaders.cc:157): Failed while visiting tablets in sys catalog: Missing table for tablet: : 04c4de3e6b5f47f3990f0a5f1218fe37
@ 0x7f23874891fc yb::LogFatalHandlerSink::send()
@ 0x7f2386669376 google::LogMessage::SendToLog()
@ 0x7f23866667da google::LogMessage::Flush()
@ 0x7f23866698a9 google::LogMessageFatal::~LogMessageFatal()
@ 0x7f2391c7b3d7 yb::master::CatalogManager::LoadSysCatalogDataTask()
@ 0x7f2387523814 yb::ThreadPool::DispatchThread()
@ 0x7f238751ffef yb::Thread::SuperviseThread()
@ 0x7f2382c48694 start_thread
@ 0x7f238238541d __clone
@ (nil) (unknown)
*** Check failure stack trace: ***
@ 0x7f23874875e1 yb::(anonymous namespace)::DumpStackTraceAndExit()
@ 0x7f2386666c8d google::LogMessage::Fail()
@ 0x7f2386668dfd google::LogMessage::SendToLog()
@ 0x7f23866667da google::LogMessage::Flush()
@ 0x7f23866698a9 google::LogMessageFatal::~LogMessageFatal()
@ 0x7f2391c7b3d7 yb::master::CatalogManager::LoadSysCatalogDataTask()
@ 0x7f2387523814 yb::ThreadPool::DispatchThread()
@ 0x7f238751ffef yb::Thread::SuperviseThread()
@ 0x7f2382c48694 start_thread
@ 0x7f238238541d __clone
@ (nil) (unknown)
Error: Failed to load sys catalog: Corruption (yb/master/catalog_loaders.cc:157): Failed while visiting tablets in sys catalog: Missing table for tablet: : 04c4de3e6b5f47f3990f0a5f1218fe37

[nspiegelberg]

Repro'd:

yugabyte=# create database bogdan with colocated=true;
yugabyte=# \c bogdan
You are now connected to database "bogdan" as user "yugabyte".
bogdan=# create table foo (k int primary key);
CREATE TABLE
bogdan=# drop table foo;
DROP TABLE

then restart the cluster

[3:04 PM] F1016 22:02:47.142489 23396 catalog_manager.cc:695] Failed to load sys catalog: Corruption (yb/master/catalog_loaders.cc:157): Failed while visiting tablets in sys catalog: Missing table for tablet: : 2e9bbcdd7a4840ba83a4f1f41a1985c7