cockroach zone ls sometimes returns error #24154

robert-s-lee · 2018-03-22T18:00:26Z

BUG REPORT

Please supply the header (i.e. the first few lines) of your most recent
log file for each node in your cluster. On most unix-based systems
running with defaults, this boils down to the output of

cockroach version
Build Tag: v2.0-beta.20180319
Build Time: 2018/03/19 18:54:14
Distribution: CCL
Platform: darwin amd64
Go Version: go1.10
C Compiler: 4.2.1 Compatible Clang 3.8.0 (tags/RELEASE_380/final)
Build SHA-1: e0b123c
Build Type: release

Please describe the issue you observed:

start from a brand new database
below sequence works

cockroach sql --insecure <<EOF
create database if not exists test;
set database=test;
drop table if exists users;
CREATE TABLE IF NOT EXISTS users (loc STRING, id INT, PRIMARY KEY (loc, id)) PARTITION BY LIST (loc) (
    PARTITION w VALUES IN ('w'),
    PARTITION e VALUES IN ('e')
);
EOF
cockroach zone ls --insecure

cockroach sql --insecure <<EOF
create database if not exists test;
set database=test;
drop table if exists users;
CREATE TABLE IF NOT EXISTS users (loc STRING, id INT, PRIMARY KEY (loc, id)) PARTITION BY LIST (loc) (
    PARTITION w VALUES IN ('eu-west-1'),
    PARTITION e VALUES IN ('eu-east-1')
);
EOF
cockroach zone ls --insecure

below works as well

cockroach sql --insecure <<EOF
create database if not exists test;
set database=test;
drop table if exists users;
CREATE TABLE IF NOT EXISTS users (loc STRING, id INT, PRIMARY KEY (loc, id)) PARTITION BY LIST (loc) (
    PARTITION w VALUES IN ('eu-west-1'),
    PARTITION e VALUES IN ('eu-east-1')
);
INSERT INTO users SELECT 'w', generate_series(1, 1000);
INSERT INTO users SELECT 'e', generate_series(1, 1000);
ALTER TABLE users SPLIT AT SELECT 'w', generate_series(1, 1000, 100);
ALTER TABLE users SPLIT AT SELECT 'e', generate_series(1, 1000, 100);
ALTER PARTITION w OF TABLE users EXPERIMENTAL CONFIGURE ZONE 'constraints: {"+region=west": 2}';
ALTER PARTITION e OF TABLE users EXPERIMENTAL CONFIGURE ZONE 'constraints: {"+region=east": 2}';
EOF
cockroach zone ls --insecure

this does not work. However, if the previous steps were not there, then this does work

cockroach sql --insecure <<EOF
create database if not exists test;
set database=test;
drop table if exists users;
CREATE TABLE IF NOT EXISTS users (loc STRING, id INT, PRIMARY KEY (loc, id)) PARTITION BY LIST (loc) (
    PARTITION w VALUES IN ('w'),
    PARTITION e VALUES IN ('e')
);
INSERT INTO users SELECT 'w', generate_series(1, 1000);
INSERT INTO users SELECT 'e', generate_series(1, 1000);
ALTER TABLE users SPLIT AT SELECT 'w', generate_series(1, 1000, 100);
ALTER TABLE users SPLIT AT SELECT 'e', generate_series(1, 1000, 100);
ALTER PARTITION w OF TABLE users EXPERIMENTAL CONFIGURE ZONE 'constraints: {"+region=west": 2}';
ALTER PARTITION e OF TABLE users EXPERIMENTAL CONFIGURE ZONE 'constraints: {"+region=east": 2}';
EOF
cockroach zone ls --insecure

$ cockroach zone ls --insecure
Error: unexpected value: <nil>
Failed running "zone"

cockroach.mb-71.rslee.2018-03-22T17_54_38Z.045302.log.gz

The text was updated successfully, but these errors were encountered:

a-robinson · 2018-03-22T18:09:08Z

I'm not sure yet whether it's related, but @vivekmenezes what's with all these errors in the logs?

W180322 17:57:10.001289 33 sql/schema_changer.go:1146  [n1] Error executing schema change: not hit TTL deadline: -24h59m29.270231106s remaining

a-robinson · 2018-03-22T19:23:21Z

Soooo it appears as though we don't delete the corresponding zone configs when we drop a table. This appears to apply to zone configs on tables as well as zone configs on a table's partitions.

Not deleting the zone configs isn't the end of the world. It might even be important if, for example, the data in the table needs to be in a certain region for regulatory reasons. In such cases it could be disastrous if the system thought it was ok to move the data around after the table is dropped but before it's GC'ed/compacted.

However:

We should filter out / stop displaying the zone configs for dropped tables/databases/partitions, since they break the command and can't be displayed properly.
It appears as though we actually do delete the corresponding zone config when dropping a database, but not a table. This is a bad inconsistency.

Any thoughts on this, @benesch?

benesch · 2018-03-22T19:29:40Z

We're definitely supposed to be dropping zone configs for tables/partitions when the table/partition they belong to is removed. Perhaps there's a bug while the table is marked as dropped but the descriptor is not yet removed?

a-robinson · 2018-03-22T19:32:57Z

Perhaps there's a bug while the table is marked as dropped but the descriptor is not yet removed?

Maybe, but if so they sure stick around for a long time. I've created half a dozen zone configs for various tables and partitions that I've since deleted, and they're all still there, even about half an hour after deleting the first one.

root@:26257/> EXPERIMENTAL SHOW ZONE CONFIGURATIONS;
+----+---------------+--------------------------------+--------------------------------------------------------------------------+
| id | cli_specifier |          config_yaml           |                               config_proto                               |
+----+---------------+--------------------------------+--------------------------------------------------------------------------+
|  0 | .default      | range_min_bytes: 1048576       | \x10\x80\x80@\x18\x80\x80\x80 \"\x04\b\x90\xbf\x05(\x03                  |
|    |               |                                |                                                                          |
|    |               | range_max_bytes: 67108864      |                                                                          |
|    |               |                                |                                                                          |
|    |               | gc:                            |                                                                          |
|    |               |                                |                                                                          |
|    |               |   ttlseconds: 90000            |                                                                          |
|    |               |                                |                                                                          |
|    |               | num_replicas: 3                |                                                                          |
|    |               |                                |                                                                          |
|    |               | constraints: []                |                                                                          |
|    |               |                                |                                                                          |
|    |               |                                |                                                                          |
| 15 | system.jobs   | range_min_bytes: 1048576       | \x10\x80\x80@\x18\x80\x80\x80 \"\x03\b\xd8\x04(\x03                      |
|    |               |                                |                                                                          |
|    |               | range_max_bytes: 67108864      |                                                                          |
|    |               |                                |                                                                          |
|    |               | gc:                            |                                                                          |
|    |               |                                |                                                                          |
|    |               |   ttlseconds: 600              |                                                                          |
|    |               |                                |                                                                          |
|    |               | num_replicas: 3                |                                                                          |
|    |               |                                |                                                                          |
|    |               | constraints: []                |                                                                          |
|    |               |                                |                                                                          |
|    |               |                                |                                                                          |
| 16 | .meta         | range_min_bytes: 1048576       | \x10\x80\x80@\x18\x80\x80\x80 \"\x03\b\x90\x1c(\x03                      |
|    |               |                                |                                                                          |
|    |               | range_max_bytes: 67108864      |                                                                          |
|    |               |                                |                                                                          |
|    |               | gc:                            |                                                                          |
|    |               |                                |                                                                          |
|    |               |   ttlseconds: 3600             |                                                                          |
|    |               |                                |                                                                          |
|    |               | num_replicas: 3                |                                                                          |
|    |               |                                |                                                                          |
|    |               | constraints: []                |                                                                          |
|    |               |                                |                                                                          |
|    |               |                                |                                                                          |
| 22 | .liveness     | range_min_bytes: 1048576       | \x10\x80\x80@\x18\x80\x80\x80 \"\x03\b\xd8\x04(\x03                      |
|    |               |                                |                                                                          |
|    |               | range_max_bytes: 67108864      |                                                                          |
|    |               |                                |                                                                          |
|    |               | gc:                            |                                                                          |
|    |               |                                |                                                                          |
|    |               |   ttlseconds: 600              |                                                                          |
|    |               |                                |                                                                          |
|    |               | num_replicas: 3                |                                                                          |
|    |               |                                |                                                                          |
|    |               | constraints: []                |                                                                          |
|    |               |                                |                                                                          |
|    |               |                                |                                                                          |
| 53 | NULL          | range_min_bytes: 1048576       | \x10\x80\x80@\x18\x80\x80\x80                                            |
|    |               |                                | \"\x04\b\x90\xbf\x05(\x032\x142\x10\b\x01\x12\x06region\x1a\x04west8\x02 |
|    |               | range_max_bytes: 67108864      |                                                                          |
|    |               |                                |                                                                          |
|    |               | gc:                            |                                                                          |
|    |               |                                |                                                                          |
|    |               |   ttlseconds: 90000            |                                                                          |
|    |               |                                |                                                                          |
|    |               | num_replicas: 3                |                                                                          |
|    |               |                                |                                                                          |
|    |               | constraints: {+region=west: 2} |                                                                          |
|    |               |                                |                                                                          |
|    |               |                                |                                                                          |
| 53 | NULL          | range_min_bytes: 1048576       | \x10\x80\x80@\x18\x80\x80\x80                                            |
|    |               |                                | \"\x04\b\x90\xbf\x05(\x032\x142\x10\b\x01\x12\x06region\x1a\x04east8\x02 |
|    |               | range_max_bytes: 67108864      |                                                                          |
|    |               |                                |                                                                          |
|    |               | gc:                            |                                                                          |
|    |               |                                |                                                                          |
|    |               |   ttlseconds: 90000            |                                                                          |
|    |               |                                |                                                                          |
|    |               | num_replicas: 3                |                                                                          |
|    |               |                                |                                                                          |
|    |               | constraints: {+region=east: 2} |                                                                          |
|    |               |                                |                                                                          |
|    |               |                                |                                                                          |
| 54 | NULL          | range_min_bytes: 1048576       | \x10\x80\x80@\x18\x80\x80\x80                                            |
|    |               |                                | \"\x04\b\x90\xbf\x05(\x032\x142\x10\b\x01\x12\x06region\x1a\x04west8\x02 |
|    |               | range_max_bytes: 67108864      |                                                                          |
|    |               |                                |                                                                          |
|    |               | gc:                            |                                                                          |
|    |               |                                |                                                                          |
|    |               |   ttlseconds: 90000            |                                                                          |
|    |               |                                |                                                                          |
|    |               | num_replicas: 3                |                                                                          |
|    |               |                                |                                                                          |
|    |               | constraints: {+region=west: 2} |                                                                          |
|    |               |                                |                                                                          |
|    |               |                                |                                                                          |
| 54 | NULL          | range_min_bytes: 1048576       | \x10\x80\x80@\x18\x80\x80\x80                                            |
|    |               |                                | \"\x04\b\x90\xbf\x05(\x032\x142\x10\b\x01\x12\x06region\x1a\x04east8\x02 |
|    |               | range_max_bytes: 67108864      |                                                                          |
|    |               |                                |                                                                          |
|    |               | gc:                            |                                                                          |
|    |               |                                |                                                                          |
|    |               |   ttlseconds: 90000            |                                                                          |
|    |               |                                |                                                                          |
|    |               | num_replicas: 3                |                                                                          |
|    |               |                                |                                                                          |
|    |               | constraints: {+region=east: 2} |                                                                          |
|    |               |                                |                                                                          |
|    |               |                                |                                                                          |
| 55 | NULL          | range_min_bytes: 1048576       | \x10\x80\x80@\x18\x80\x80\x80                                            |
|    |               |                                | \"\x04\b\x90\xbf\x05(\x032\x142\x10\b\x01\x12\x06region\x1a\x04alex8\x02 |
|    |               | range_max_bytes: 67108864      |                                                                          |
|    |               |                                |                                                                          |
|    |               | gc:                            |                                                                          |
|    |               |                                |                                                                          |
|    |               |   ttlseconds: 90000            |                                                                          |
|    |               |                                |                                                                          |
|    |               | num_replicas: 3                |                                                                          |
|    |               |                                |                                                                          |
|    |               | constraints: {+region=alex: 2} |                                                                          |
|    |               |                                |                                                                          |
|    |               |                                |                                                                          |
| 58 | NULL          | range_min_bytes: 1048576       | \x10\x80\x80@\x18\x80\x80\x80                                            |
|    |               |                                | \"\x04\b\x90\xbf\x05(\x032\x142\x10\b\x01\x12\x06region\x1a\x04alex8\x03 |
|    |               | range_max_bytes: 67108864      |                                                                          |
|    |               |                                |                                                                          |
|    |               | gc:                            |                                                                          |
|    |               |                                |                                                                          |
|    |               |   ttlseconds: 90000            |                                                                          |
|    |               |                                |                                                                          |
|    |               | num_replicas: 3                |                                                                          |
|    |               |                                |                                                                          |
|    |               | constraints: {+region=alex: 3} |                                                                          |
|    |               |                                |                                                                          |
|    |               |                                |                                                                          |
+----+---------------+--------------------------------+--------------------------------------------------------------------------+

benesch · 2018-03-22T19:50:03Z

cockroach/pkg/sql/schema_changer.go

Lines 274 to 277 in 586afb0

    
           // Delete the zone config entry for this table. 
        
           b.DelRange(zoneKeyPrefix, zoneKeyPrefix.PrefixEnd(), false /* returnKeys */) 
        
           if err := txn.SetSystemConfigTrigger(); err != nil { 
        
           	return err

benesch · 2018-03-22T19:51:03Z

Per the above, your GC TTL is set to the default 25h, so that's not unexpected.

a-robinson · 2018-03-22T19:59:39Z

Ok, so we really just need to hide / ignore the NULL cli_specifier field for these in the zone ls output, assuming that holds.

It's pretty inconsistent that we drop a database's zone config immediately, though. If a table's replication is being controlled by the database-level zone config, when you run DROP DATABASE ... CASCADE the data in the table won't be constrained in the same way as if the table's replication was controlled by a table-level zone config.

benesch · 2018-03-22T20:08:31Z

Ok, so we really just need to hide / ignore the NULL cli_specifier field for these in the zone ls output, assuming that holds.

Yep, think so.

It's pretty inconsistent that we drop a database's zone config immediately, though. If a table's replication is being controlled by the database-level zone config, when you run DROP DATABASE ... CASCADE the data in the table won't be constrained in the same way as if the table's replication was controlled by a table-level zone config.

Yikes, definitely.

If I seem curt, it's only because I'm semi-AFK! Thanks for looking into this, Alex.

This continues displaying them in the output of `SHOW EXPERIMENTAL ZONE CONFIGURATIONS`, but it still makes some sense there given that the zone configs are still in use, they just aren't addressable. Fixes cockroachdb#24154 Release note (bug fix): Fix a crash in `cockroach zone ls` that would happen if a table with a zone config on it had been deleted but not yet garbage collected. (this was broken in v2.0 alphas, not in v1.1)

a-robinson added this to the 2.0 milestone Mar 22, 2018

a-robinson self-assigned this Mar 22, 2018

This was referenced Mar 23, 2018

cli: Don't fail zone ls on zones from deleted tables/partitions #24178

Merged

sql: Inconsistent zone config behavior when dropping database or table #24179

Closed

a-robinson mentioned this issue Mar 23, 2018

cherrypick-2.0: cli: Don't fail zone ls on zones from deleted tables/partitions #24180

Merged

a-robinson closed this as completed in #24178 Mar 23, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cockroach zone ls sometimes returns error #24154

cockroach zone ls sometimes returns error #24154

robert-s-lee commented Mar 22, 2018

a-robinson commented Mar 22, 2018

a-robinson commented Mar 22, 2018 •

edited

Loading

benesch commented Mar 22, 2018

a-robinson commented Mar 22, 2018 •

edited

Loading

benesch commented Mar 22, 2018

benesch commented Mar 22, 2018

a-robinson commented Mar 22, 2018 •

edited

Loading

benesch commented Mar 22, 2018

cockroach zone ls sometimes returns error #24154

cockroach zone ls sometimes returns error #24154

Comments

robert-s-lee commented Mar 22, 2018

a-robinson commented Mar 22, 2018

a-robinson commented Mar 22, 2018 • edited Loading

benesch commented Mar 22, 2018

a-robinson commented Mar 22, 2018 • edited Loading

benesch commented Mar 22, 2018

benesch commented Mar 22, 2018

a-robinson commented Mar 22, 2018 • edited Loading

benesch commented Mar 22, 2018

a-robinson commented Mar 22, 2018 •

edited

Loading

a-robinson commented Mar 22, 2018 •

edited

Loading

a-robinson commented Mar 22, 2018 •

edited

Loading