Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sql/multitenancy: alter_primary_key fails with unsuccesful job #52558

Closed
Tracked by #52617
asubiotto opened this issue Aug 10, 2020 · 3 comments · Fixed by #52984
Closed
Tracked by #52617

sql/multitenancy: alter_primary_key fails with unsuccesful job #52558

asubiotto opened this issue Aug 10, 2020 · 3 comments · Fixed by #52984
Assignees
Labels
A-multitenancy Related to multi-tenancy C-test-failure Broken test (automatically or manually discovered).

Comments

@asubiotto
Copy link
Contributor

Running alter_primary_key with 3node-tenant results in the following error:

    --- FAIL: TestCCLLogic/3node-tenant (0.00s)
        --- FAIL: TestCCLLogic/3node-tenant/__test (2.25s)
            logic.go:2308:

                testdata/logic_test/__test:195:
                expected success, but found
                (XXUUU) failed to mark job 579940132452869905 as as successful: Job with status failed cannot be marked as succeeded
                jobs.go:587: in func1()
            logic.go:2644:
                testdata/logic_test/__test:195: error while processing
            logic.go:2644: pq: failed to mark job 579940132452869905 as as successful: Job with status failed cannot be marked as succeeded
FAIL
@asubiotto asubiotto added the A-multitenancy Related to multi-tenancy label Aug 10, 2020
@asubiotto asubiotto self-assigned this Aug 13, 2020
@asubiotto asubiotto added the C-test-failure Broken test (automatically or manually discovered). label Aug 18, 2020
@cockroachdb cockroachdb deleted a comment from blathers-crl bot Aug 18, 2020
@asubiotto
Copy link
Contributor Author

Minimal reproduction:

# LogicTest: 3node-tenant

# Test that we can change the primary key of an interleaved child.
statement ok
CREATE TABLE parent (x INT, y INT, PRIMARY KEY (x, y), FAMILY (x, y));
CREATE TABLE child (
  x INT, y INT NOT NULL, z INT NOT NULL,
  FAMILY (x, y, z), PRIMARY KEY (x, y, z)
) INTERLEAVE IN PARENT parent (x, y);
ALTER TABLE child ALTER PRIMARY KEY USING COLUMNS (y, z)

# We can drop parent because it isn't an interleaved parent anymore.
statement ok
DROP TABLE parent

statement ok
DROP TABLE IF EXISTS child;

Can be run by make test PKG=./pkg/ccl/logictestccl TESTS=TestTenantLogic//__test where __test is a file in pkg/sql/logictest/tesdata/logic_test with the above output.

@ajwerner would you mind taking a quick look? I'm not sure how to figure out what job failed and why and whether drop table should be concerning itself with these failed jobs.

cc @rohany since it touches on primary key changes

@rohany
Copy link
Contributor

rohany commented Aug 18, 2020

I see this stack:

I200818 14:22:14.955318 4037 jobs/registry.go:988  [sql] SCHEMA CHANGE job 582255034000090897: stepping through state failed with error: job 582255034000090897: cannot be reverted, manual cleanup may be required: duplicate constraint name: "primary"
(1) attached stack trace
  | github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine
  | 	/Users/rohany/go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1083
  | [...repeated from below...]
Wraps: (2) 2 safe details enclosed
Wraps: (3) job 582255034000090897: cannot be reverted, manual cleanup may be required
Wraps: (4) secondary error attachment
  | original error when rolling back mutations: operation is unsupported in multi-tenancy mode; see:
  | (1) attached stack trace
  |   | github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChanger).handlePermanentSchemaChangeError
  |   | 	/Users/rohany/go/src/github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:680
  |   | github.com/cockroachdb/cockroach/pkg/sql.schemaChangeResumer.OnFailOrCancel
  |   | 	/Users/rohany/go/src/github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:2080
  |   | github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine
  |   | 	/Users/rohany/go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1057
  |   | github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine
  |   | 	/Users/rohany/go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1021
  |   | github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).runJob
  |   | 	/Users/rohany/go/src/github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:202
  |   | github.com/cockroachdb/cockroach/pkg/jobs.(*StartableJob).Start.func2
  |   | 	/Users/rohany/go/src/github.com/cockroachdb/cockroach/pkg/jobs/jobs.go:898
  |   | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1
  |   | 	/Users/rohany/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:347
  |   | runtime.goexit
  |   | 	/usr/local/Cellar/[email protected]/1.13.15/libexec/src/runtime/asm_amd64.s:1357
  | Wraps: (2) 1 safe detail enclosed
  | Wraps: (3) original error when rolling back mutations
  | Wraps: (4)
  |   | (opaque error wrapper)
  |   | type name: github.com/cockroachdb/errors/withstack/*withstack.withStack
  |   | reportable 0:
  |   |
  |   | github.com/cockroachdb/cockroach/pkg/util/errorutil.UnsupportedWithMultiTenancy
  |   | 	/Users/rohany/go/src/github.com/cockroachdb/cockroach/pkg/util/errorutil/tenant.go:36
  |   | github.com/cockroachdb/cockroach/pkg/sql.writeZoneConfig
  |   | 	/Users/rohany/go/src/github.com/cockroachdb/cockroach/pkg/sql/set_zone_config.go:850
  |   | github.com/cockroachdb/cockroach/pkg/sql.RemoveIndexZoneConfigs
  |   | 	/Users/rohany/go/src/github.com/cockroachdb/cockroach/pkg/sql/set_zone_config.go:934
  |   | github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChanger).truncateIndexes.func2
  |   | 	/Users/rohany/go/src/github.com/cockroachdb/cockroach/pkg/sql/backfill.go:756
  |   | github.com/cockroachdb/cockroach/pkg/kv.(*DB).Txn.func1
  |   | 	/Users/rohany/go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:707
  |   | github.com/cockroachdb/cockroach/pkg/kv.(*Txn).exec
  |   | 	/Users/rohany/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn.go:811
  |   | github.com/cockroachdb/cockroach/pkg/kv.(*DB).Txn
  |   | 	/Users/rohany/go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:706
  |   | github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChanger).truncateIndexes
  |   | 	/Users/rohany/go/src/github.com/cockroachdb/cockroach/pkg/sql/backfill.go:755
  |   | github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChanger).runBackfill
  |   | 	/Users/rohany/go/src/github.com/cockroachdb/cockroach/pkg/sql/backfill.go:264
  |   | github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChanger).runStateMachineAndBackfill
  |   | 	/Users/rohany/go/src/github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:1337
  |   | github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChanger).exec
  |   | 	/Users/rohany/go/src/github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:645
  |   | github.com/cockroachdb/cockroach/pkg/sql.schemaChangeResumer.Resume.func1
  |   | 	/Users/rohany/go/src/github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:1951
  |   | github.com/cockroachdb/cockroach/pkg/sql.schemaChangeResumer.Resume
  |   | 	/Users/rohany/go/src/github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:2035
  |   | github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine
  |   | 	/Users/rohany/go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:996
  |   | github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).runJob
  |   | 	/Users/rohany/go/src/github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:202
  |   | github.com/cockroachdb/cockroach/pkg/jobs.(*StartableJob).Start.func2
  |   | 	/Users/rohany/go/src/github.com/cockroachdb/cockroach/pkg/jobs/jobs.go:898
  |   | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1
  |   | 	/Users/rohany/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:347
  |   | runtime.goexit
  |   | 	/usr/local/Cellar/[email protected]/1.13.15/libexec/src/runtime/asm_amd64.s:1357
  | Wraps: (5) 2 safe details enclosed
  | Wraps: (6) operation is unsupported in multi-tenancy mode; see:

It looks like the truncateIndexes call needs a isForSystemTenant check or something

@asubiotto
Copy link
Contributor Author

Thanks for spotting that!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-multitenancy Related to multi-tenancy C-test-failure Broken test (automatically or manually discovered).
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants