sql: database exists in SHOW DATABASE list after DROP

[nvanbenschoten]

@HeikoOnnebrink (from #17524) recently ran into some strangeness around dropping tables.
He dropped two tables but they continued to show up in SHOW TABLES queries. If he issued
further DROP TABLE requests, he would get an error pq: relation "requests" does not exist.

Looking quickly through his logs, this line stood out:

W170906 05:37:47.344467 166 sql/schema_changer.go:1072 Error executing schema change: descriptor not found

Later he tried to DROP DATABASE. Even after he issued this, the database remained in his
SHOW DATABASES queries. However, when he tries to do anything in the database,
he gets an error:

pq: no database with ID 52 found

If we want to investigate the corrupted state, we have access to this cluster via a Cisco WebEx session (you'll need a chrome plugin) at a meeting link that can be activated by Heiko on Gitter (he'll supervise while you are connected).

[nvanbenschoten]

@vivekmenezes for triage.

[HeikoOnnebrink]

short adjustment ..
I issued several DROP TABLE commands on 6 tables... a 6 gave positive feedback, but 2 tables nonetheless still showed up in SHOW TABLES.. Accessing any of these tables returned pq: relation "xxx" does not exist
To overcome this problem I tried a DELETE DATABASE which also worked, but the logs filled up with
E170906 05:52:31.802781 372702 server/admin.go:519 database "restapipp" does not exist
After restarting cluster the database disappeared from the list of databases, but problems persist

[email protected]:26257/restapi> show databases;
+--------------------+
| Database |
+--------------------+
| crdb_internal |
| information_schema |
| pg_catalog |
| restapi |
+--------------------+
(4 rows)
[email protected]:26257/restapi> show tables;
pq: no database with ID 52 found
[email protected]:26257/restapi> show columns from postgresqldbs;
pq: no database with ID 52 found

[jordanlewis]

Closing due to inactivity.