Table dropped only partially

[arctica]

Running v1.2-alpha.20170901-801-g5282759 on a 3 node cluster. I ran the load test from #19236 and issued a DROP TABLE test.bench; while the load generator was still running. This seemed to hang or take a long time so I cancelled this statement via ctrl-c in the cli. Then I stopped the load generator. I saw the table disappeared and I could create a new table.

The admin UI though indicates that the data of the old table is still there. Disk storage is still being used, the Replicas graph still shows 150k replicas and the Replication page shows 150k ranges while the new table has only a few dozen.

[arctica]

Thinking about it, might this have to do with the time travel support in CockroachDB? Does it keep the data around for 24h and if so, is it possible to get the table back?

[vivekmenezes]

There are three layers of activity going in in DROP.

1. The DROP command itself which once completes guarantees that a user cannot access the data any more and the schema is gone
2. the data delete step that starts off in about 5 minutes after DROP that starts deleting the data.
3. The GC step that runs after 24 hours that completely removes the data and recovers the underlying storage for the user.

The only support to get the table back in 1.1 is to use a time travel query with a timestamp before the DROP and funnel the output of that query into a new table.

[a-robinson]

Thinking about it, might this have to do with the time travel support in CockroachDB? Does it keep the data around for 24h and if so, is it possible to get the table back?

Yes, that's why the storage usage hasn't dropped. Although even after the 25 hour TTL, it could still take a little longer to truly get cleared off the disk due to how deletions in RocksDB (our underlying embedded KV store) work. #19329 was recently filed to allow for speeding this up when necessary.

[a-robinson]

the Replicas graph still shows 150k replicas and the Replication page shows 150k ranges while the new table has only a few dozen.

This part won't change, though. The data in the ranges will get GC'ed, and the ranges will "quiesce", but they'll still exist. We don't yet support merging small/empty ranges back together, and deleting them entirely would leave holes in the KV keyspace. Having them around shouldn't cost much, since they won't ever have anything to do and thus shouldn't take up memory or much processing power within your nodes, but they'll still be there in the graphs.

[a-robinson]

I'm going to close this as being covered by #19329 and cockroachdb/docs#2077, but reopen if there's anything you think isn't covered elsewhere!