Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

multi-tenant: enable IMPORT for workload init in multi-tenant mode #75449

Closed
ajstorm opened this issue Jan 24, 2022 · 0 comments · Fixed by #78418
Closed

multi-tenant: enable IMPORT for workload init in multi-tenant mode #75449

ajstorm opened this issue Jan 24, 2022 · 0 comments · Fixed by #78418
Labels
A-multitenancy Related to multi-tenancy C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)

Comments

@ajstorm
Copy link
Collaborator

ajstorm commented Jan 24, 2022 •
edited by cockroach-jira-scripts
Loading

Currently workload init does not support IMPORT as a data loader option when running in multi-tenant mode. Instead, the following error gets returned:

 cockroach workload init tpcc --warehouses=10 --data-loader=IMPORT --partitions=3 --survival-goal zone --regions=europe-west1,us-east1,us-west1 'postgresql://[email protected]:26257/tpcc?sslmode=disable'

I220124 16:39:50.808709 1 ccl/workloadccl/fixture.go:345  [-] 1  starting import of 9 tables
Error: importing fixture: pq: unimplemented: operation is unsupported in multi-tenancy mode

This error is being triggered by the fixture code, which is trying to determine how much to parallelize the IMPORT, using a query of the gossip_liveness table, which is not visible to tenants.

cockroach/pkg/ccl/workloadccl/fixture.go

Line 382 in 09abc26

if err := sqlDB.QueryRow(numNodesQuery).Scan(&numNodes); err != nil {

Short term we can hard-code the numNodes here to 1, as we don't support DistSQL for IMPORTs in multi-tenant mode. Longer term however, we'll want an API which allows tenants to query the number of DistSQL pods available for these types of IMPORT jobs.

Jira issue: CRDB-12677
@ajstorm ajstorm added C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) A-multitenancy Related to multi-tenancy labels Jan 24, 2022
@ajstorm ajstorm mentioned this issue Jan 31, 2022
Open
@ajstorm ajstorm mentioned this issue Feb 10, 2022
Open
This was referenced Mar 29, 2022
Merged
Closed
craig bot pushed a commit that referenced this issue Mar 29, 2022
@nvanbenschoten @msbutler
@RichardJCai @mgartner @adityamaru @aayushshah15
Merge #78418 #78626 #78653 #78685 #78940 #78962 #78971
897c2da 
78418: workload: support fixture import in multi-tenant mode r=nvanbenschoten a=nvanbenschoten

Fixes #75449.

This commit fixes fixture import into tenants, which do not have access to the `crdb_internal.gossip_liveness` table. This fixes the following:
- `[cockroach] workload fixtures import`
- `[cockroach] workload init --data-loader=IMPORT`

Release justification: low-risk change that improves workload functionality.

78626: backupccl: add deprecation warning for SHOW BACKUP without subdir r=adityamaru,benbardin a=msbutler

Informs: #78153

Release note (sql change): The `SHOW BACKUP` cmd  without the `IN` keyword to
specify a subdirectory is deprecated and will be removed from a future release.
Users are recommended to only create collection based backups and view them
with `SHOW BACKUP FROM <backup> IN <collection>`

78653: sql: add logging for grant_role r=RichardJCai a=RichardJCai

GRANT ROLE now logs to the logpb.Channel_USER_ADMIN channel.
Example: GRANT role1, role2 TO role3, role4 logs
"RoleNames":"‹role1›",‹role2›","Members":"‹role3›",‹ role4›"

Release note: None

Fixes #78245

78685: opt: do not cross-join input of semi-join r=mgartner a=mgartner

This commit fixes a logical correctness bug caused when
`GenerateLookupJoins` cross-joins the input of a semi-join with a set of
constant values to constrain the prefix columns of the lookup index. The
cross-join is an invalid transformation because it increases the size of
the join's input and can increase the size of the join's output.

We already avoid these cross-joins for left and anti-joins (see #59646).
When addressing those cases, the semi-join case was incorrectly assumed
to be safe.

Fixes #78681

Release note (bug fix): A bug has been fixed which caused the optimizer
to generate invalid query plans which could result in incorrect query
results. The bug, which has been present since version 21.1.0, can
appear if all of the following conditions are true: 1) the query
contains a semi-join, such as queries in the form:
`SELECT * FROM t1 WHERE EXISTS (SELECT * FROM t2 WHERE t1.a = t2.a);`,
2) the inner table has an index containing the equality column, like
`t2.a` in the example query, 3) the index contains one or more
columns that prefix the equality column, and 4) the prefix columns are
`NOT NULL` and are constrained to a set of constant values via a `CHECK`
constraint or an `IN` condition in the filter.


78940: backupccl: fix nil pointer exception in restore OnFailOrCancel r=dt,stevendanna a=adityamaru

The execCfg field on the restore resumer was not being set correctly
in the OnFailOrCancel hook. This would lead to a nil pointer exception
during cleanup.

Other jobs such as import, backup use the execCfg directly off the
JobExecContext instead of storing it on the resumer. An alternative to
this fix could be to change all `r.ExecCfg` to use the execCfg on the JobExecContext.

Fixes: #76720

Release note (bug fix): Fixes a nil pointer exception during the
cleanup of a failed or cancelled restore job.

78962: kvserver: make the StoreRebalancer interval a cluster setting r=aayushshah15 a=aayushshah15

Release note (ops change): the `kv.allocator.load_based_rebalancing_interval`
cluster setting now lets operators choose the interval at which each store in the
cluster will check for load-based lease or replica rebalancing opportunities.


78971: sql, migration: skip flaky TestPublicSchemaMigration test r=ajwerner a=RichardJCai

Release note: None

Co-authored-by: Nathan VanBenschoten <[email protected]>
Co-authored-by: Michael Butler <[email protected]>
Co-authored-by: richardjcai <[email protected]>
Co-authored-by: Marcus Gartner <[email protected]>
Co-authored-by: Aditya Maru <[email protected]>
Co-authored-by: Aayush Shah <[email protected]>
@craig craig bot closed this as completed in #78418 Mar 29, 2022
@craig craig bot closed this as completed in 11d9b3d Mar 29, 2022
blathers-crl bot pushed a commit that referenced this issue Mar 29, 2022
@nvanbenschoten
workload: support fixture import in multi-tenant mode
2955319 
Fixes #75449.

This commit fixes fixture import into tenants, which do not have access
to the `crdb_internal.gossip_liveness` table. This fixes the following:
- `[cockroach] workload fixtures import`
- `[cockroach] workload init --data-loader=IMPORT`

Release justification: low-risk change that improves workload functionality.
@blathers-crl blathers-crl bot mentioned this issue Mar 29, 2022
Closed
nvanbenschoten added a commit to nvanbenschoten/cockroach that referenced this issue Apr 4, 2022
@nvanbenschoten
workload: support fixture import in multi-tenant mode
d23fa90 
Fixes cockroachdb#75449.

This commit fixes fixture import into tenants, which do not have access
to the `crdb_internal.gossip_liveness` table. This fixes the following:
- `[cockroach] workload fixtures import`
- `[cockroach] workload init --data-loader=IMPORT`

Release justification: low-risk change that improves workload functionality.
@nvanbenschoten nvanbenschoten mentioned this issue Apr 4, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
A-multitenancy Related to multi-tenancy C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

workload: support fixture import in multi-tenant mode nvanbenschoten/cockroach
1 participant
@ajstorm