Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

distsql: testing infrastructure #13665

Closed
RaduBerinde opened this issue Feb 18, 2017 · 7 comments
Closed

distsql: testing infrastructure #13665

RaduBerinde opened this issue Feb 18, 2017 · 7 comments
Assignees
@RaduBerinde
Milestone
1.1

Comments

@RaduBerinde
Copy link
Member

RaduBerinde commented Feb 18, 2017 •
edited
Loading

This issue tracks adding better testing infrastructure for DistSQL. In a gist, we want to be able to easily set up tests like those in TestDistSQLPlanner from logic test files.
Once available, we should switch over these tests to the new infrastructure.

The two major things we want:

  • a way to split and distribute tables the way we want; can augment SPLIT AT with replica information for the new split and/or use a special column in tables to determine split points. Also switch SPLIT AT to a function so that we can feed it from a subquery.
  • a way to generate table data procedurally; we may be able to use subqueries with generators (GENERATE_SERIES) but we at least need to develop some sample patterns.

CC @andreimatei
@RaduBerinde RaduBerinde mentioned this issue Feb 18, 2017
Closed
18 tasks
@cuongdo cuongdo added this to the Q1 2017 milestone Feb 22, 2017
@RaduBerinde
Copy link
Member Author

I can work on this in the following days.

@RaduBerinde RaduBerinde self-assigned this Mar 19, 2017
RaduBerinde added a commit to RaduBerinde/cockroach that referenced this issue Mar 20, 2017
@RaduBerinde
sql: change SPLIT AT to take a select statement
b39559c 
`SPLIT AT` now takes an arbitrary select statement. Existing uses must switch to
using `VALUES`; e.g. `ALTER TABLE t SPLIT AT (x, y)` becomes `ALTER TABLE t
SPLIT AT VALUES (x, y)`.

Part of cockroachdb#13665, implements part of RFC cockroachdb#14146.
@RaduBerinde RaduBerinde mentioned this issue Mar 20, 2017
Merged
RaduBerinde added a commit to RaduBerinde/cockroach that referenced this issue Mar 21, 2017
@RaduBerinde
sql: change SPLIT AT to take a select statement
40f7517 
`SPLIT AT` now takes an arbitrary select statement. Existing uses must switch to
using `VALUES`; e.g. `ALTER TABLE t SPLIT AT (x, y)` becomes `ALTER TABLE t
SPLIT AT VALUES (x, y)`.

Part of cockroachdb#13665, implements part of RFC cockroachdb#14146.
@RaduBerinde RaduBerinde mentioned this issue Mar 22, 2017
Merged
@RaduBerinde
Copy link
Member Author

RaduBerinde commented Mar 22, 2017 •
edited by jordanlewis
Loading

Checklist for implementing the RFC in #14146:

  • implementing the new SPLIT AT syntax
  • implementing the new TESTING_RELOCATE statement
  • implementing the new SCATTER statement

RaduBerinde added a commit to RaduBerinde/cockroach that referenced this issue Mar 23, 2017
@RaduBerinde
storage: plumb AdminChangeReplicas request
531f84b 
Plumbing an `AdminChangeReplicas` request. Refactoring the `testcluster` code
to issue this operation rather than calling into the `Replica` directly.

This request will be used to implement `SCATTER` and `TESTING_RELOCATE`
(cockroachdb#13665).
@RaduBerinde RaduBerinde mentioned this issue Mar 23, 2017
Merged
RaduBerinde added a commit to RaduBerinde/cockroach that referenced this issue Mar 23, 2017
@RaduBerinde
storage: plumb AdminChangeReplicas request
36b9dcb 
Plumbing an `AdminChangeReplicas` request. Refactoring the `testcluster` code
to issue this operation rather than calling into the `Replica` directly.

This request will be used to implement `SCATTER` and `TESTING_RELOCATE`
(cockroachdb#13665).
RaduBerinde added a commit to RaduBerinde/cockroach that referenced this issue Mar 23, 2017
@RaduBerinde
storage: plumb AdminChangeReplicas request
9e2fa01 
Plumbing an `AdminChangeReplicas` request. Refactoring the `testcluster` code
to issue this operation rather than calling into the `Replica` directly.

This request will be used to implement `SCATTER` and `TESTING_RELOCATE`
(cockroachdb#13665).
@RaduBerinde
Copy link
Member Author

Quoting an important suggestion by @petermattis:

To implement scatter, we could add a new admin endpoint that returned the candidate stores given a table or zone config. This would also be useful for upcoming work in the UI to show the effect of a zone config. Cc @BramGruneir, @mrtracy.

@RaduBerinde RaduBerinde mentioned this issue Mar 27, 2017
Merged
@RaduBerinde
Copy link
Member Author

SHOW TESTING_RANGES and TESTING_RELOCATE are in, these were the requirements for distsql tests. I am closing this. I don't think anything is tracking the implementation of SCATTER, @danhhz feel free to create an issue to track that.

@RaduBerinde
Copy link
Member Author

Actually, reopening to track implementation of SCATTER.

@petermattis: regarding your suggestion above: is the idea that we add this to the serverpb admin API and have the sql layer get the info via a loopback HTTP get? One problem is that serverpb depends on storage and I don't know if we want sql to depend on storage (even indirectly).

@RaduBerinde RaduBerinde reopened this Apr 9, 2017
@petermattis
Copy link
Collaborator

@petermattis: regarding your suggestion above: is the idea that we add this to the serverpb admin API and have the sql layer get the info via a loopback HTTP get? One problem is that serverpb depends on storage and I don't know if we want sql to depend on storage (even indirectly).

My idea was only half-baked. Yes, I was suggesting an admin API endpoint. But perhaps sql should have some internal call it can make, passed in via sql.ExecutorConfig.

@RaduBerinde
Copy link
Member Author

Got it. Yeah, I think that even if we have an admin endpoint for other reasons, it's probably easier if we provide an internal interface for sql to call into that implementation directly.

@RaduBerinde RaduBerinde mentioned this issue Apr 11, 2017
Merged
RaduBerinde added a commit to RaduBerinde/cockroach that referenced this issue Apr 11, 2017
@RaduBerinde
sql: basic SCATTER implementation
5fdd260 
Initial simplistic implementation of SCATTER: we pick three random stores in the
cluster for each range. The intention is to later replace this with a complete
implementation which gets replica stores recommendations according to the zone
config.

Updates cockroachdb#13665.
RaduBerinde added a commit to RaduBerinde/cockroach that referenced this issue Apr 12, 2017
@RaduBerinde
sql: basic SCATTER implementation
90533d1 
Initial simplistic implementation of SCATTER: we pick three random stores in the
cluster for each range. The intention is to later replace this with a complete
implementation which gets replica stores recommendations according to the zone
config.

Updates cockroachdb#13665.
@dianasaur323 dianasaur323 modified the milestones: Q2 2017, Q1 2017, 1.1 Apr 18, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@RaduBerinde RaduBerinde

Labels
None yet
Projects
None yet
Milestone
1.1
Development

No branches or pull requests
5 participants
@cuongdo @dianasaur323 @petermattis @asubiotto @RaduBerinde