Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

alter table of parent geo-partitioned table affects select on child #9547

Closed
kerneljake opened this issue Jul 30, 2021 · 2 comments
Closed
Assignees
Labels
area/ysql Yugabyte SQL (YSQL)

Comments

@kerneljake
Copy link
Contributor

ALTER TABLE on the parent geo-partitioned table affects SELECT on the child table.

The following example uses the schema from the documentation.

% ysqlsh
ysqlsh (11.2-YB-2.7.2.0-b0)

localhost:5433 yugabyte@yugabyte=# alter table transactions add column foo varchar(30);
ALTER TABLE

localhost:5433 yugabyte@yugabyte=# \d transactions_eu;
                        Table "public.transactions_eu"
    Column     |            Type             | Collation | Nullable | Default
---------------+-----------------------------+-----------+----------+---------
 user_id       | integer                     |           | not null |
 account_id    | integer                     |           | not null |
 geo_partition | character varying           |           | not null |
 account_type  | character varying           |           | not null |
 amount        | numeric                     |           | not null |
 txn_type      | character varying           |           | not null |
 created_at    | timestamp without time zone |           | not null | now()
 foo           | character varying(30)       |           |          |
Partition of: transactions FOR VALUES IN ('EU', 'EMEA')
Indexes:
    "transactions_eu_pkey" PRIMARY KEY, lsm ((user_id, geo_partition) HASH, account_id, created_at)
Tablespace: "eu_tablespace"

localhost:5433 yugabyte@yugabyte=# select * from transactions_eu;
ERROR:  Invalid argument: Invalid column number 8
@deeps1991
Copy link
Contributor

deeps1991 commented Aug 9, 2021

The issue is because altering the partitioned table results in adding a column to the partitioned table, and then recursively adding a column for all the child tables. This is done correctly at the PG metadata side.
However the corresponding Yugabyte Alter command call is invoked only for the parent partitioned table:

static void                                                                     
  YBCPrepareAlterTableCmd(AlterTableCmd* cmd, Relation rel, YBCPgStatement handle,
                          int* col, bool* needsYBAlter,                           
                          YBCPgStatement* rollbackHandle)                         
  {                                                                               
      Oid relationId = RelationGetRelid(rel);                                     
      switch (cmd->subtype)                                                       
      {                                                                           
          case AT_AddColumn:                                                      
          case AT_AddColumnToView:                                                
          case AT_AddColumnRecurse:                                               
          {                                                                       
              ColumnDef* colDef = (ColumnDef *) cmd->def;                         
              Oid         typeOid;                                                
              int32       typmod;                                                 
              HeapTuple   typeTuple;                                              
              int order;                                                          
                                                                                 
              typeTuple = typenameType(NULL, colDef->typeName, &typmod);          
              typeOid = HeapTupleGetOid(typeTuple);                               
              order = RelationGetNumberOfAttributes(rel) + *col;                  
              const YBCPgTypeEntity *col_type = YBCDataTypeFromOidMod(order, typeOid);
                                                                                  
              HandleYBStatus(YBCPgAlterTableAddColumn(handle, colDef->colname,    
                                                      order, col_type));          
              ++(*col);                                                           
              ReleaseSysCache(typeTuple);                                         
              *needsYBAlter = true;                  

The fix is to add a column at the YB side for all the partitions as well in the above code.

@nitin302
Copy link

nitin302 commented Feb 1, 2022

When can we expect this to be released?

deeps1991 added a commit that referenced this issue Mar 5, 2022
…ect on child

Summary:
Failing example:
```
-- Add column to parent partitioned table.
yugabyte@yugabyte=# alter table transactions add column foo varchar(30);
ALTER TABLE

-- Column is added to child partition in PG metadata.
yugabyte@yugabyte=# \d transactions_eu;
                        Table "public.transactions_eu"
    Column     |            Type             | Collation | Nullable | Default
---------------+-----------------------------+-----------+----------+---------
 user_id       | integer                     |           | not null |
 geo_partition | character varying           |           | not null |
 foo           | character varying(30)       |           |          |
Partition of: transactions FOR VALUES IN ('EU', 'EMEA')

-- Select data from child
localhost:5433 yugabyte@yugabyte=# select * from transactions_eu;
ERROR:  Invalid argument: Invalid column number 8
```

The above issue is happening because ALTER on parent partitioned table
recursively reflects the ALTER operation on the child partitions by
the PG layer. However, this does not happen on the DocDB side.
This leads to mismatch in the YSQL and DocDB metadata for the child
partition tables following an ALTER, resulting in failure of future
DQL and DML operations.

Test Plan: ybd --scb --sj --java-test org.yb.pgsql.TestPgRegressPartitions

Reviewers: myang

Reviewed By: myang

Subscribers: ena, myang, yql

Differential Revision: https://phabricator.dev.yugabyte.com/D14685
jayant07-yb pushed a commit to jayant07-yb/yugabyte-db that referenced this issue Mar 8, 2022
…ects select on child

Summary:
Failing example:
```
-- Add column to parent partitioned table.
yugabyte@yugabyte=# alter table transactions add column foo varchar(30);
ALTER TABLE

-- Column is added to child partition in PG metadata.
yugabyte@yugabyte=# \d transactions_eu;
                        Table "public.transactions_eu"
    Column     |            Type             | Collation | Nullable | Default
---------------+-----------------------------+-----------+----------+---------
 user_id       | integer                     |           | not null |
 geo_partition | character varying           |           | not null |
 foo           | character varying(30)       |           |          |
Partition of: transactions FOR VALUES IN ('EU', 'EMEA')

-- Select data from child
localhost:5433 yugabyte@yugabyte=# select * from transactions_eu;
ERROR:  Invalid argument: Invalid column number 8
```

The above issue is happening because ALTER on parent partitioned table
recursively reflects the ALTER operation on the child partitions by
the PG layer. However, this does not happen on the DocDB side.
This leads to mismatch in the YSQL and DocDB metadata for the child
partition tables following an ALTER, resulting in failure of future
DQL and DML operations.

Test Plan: ybd --scb --sj --java-test org.yb.pgsql.TestPgRegressPartitions

Reviewers: myang

Reviewed By: myang

Subscribers: ena, myang, yql

Differential Revision: https://phabricator.dev.yugabyte.com/D14685
deeps1991 added a commit that referenced this issue Mar 16, 2022
…le affects select on child

Summary:
Failing example:
```
-- Add column to parent partitioned table.
yugabyte@yugabyte=# alter table transactions add column foo varchar(30);
ALTER TABLE

-- Column is added to child partition in PG metadata.
yugabyte@yugabyte=# \d transactions_eu;
                        Table "public.transactions_eu"
    Column     |            Type             | Collation | Nullable | Default
---------------+-----------------------------+-----------+----------+---------
 user_id       | integer                     |           | not null |
 geo_partition | character varying           |           | not null |
 foo           | character varying(30)       |           |          |
Partition of: transactions FOR VALUES IN ('EU', 'EMEA')

-- Select data from child
localhost:5433 yugabyte@yugabyte=# select * from transactions_eu;
ERROR:  Invalid argument: Invalid column number 8
```

The above issue is happening because ALTER on parent partitioned table
recursively reflects the ALTER operation on the child partitions by
the PG layer. However, this does not happen on the DocDB side.
This leads to mismatch in the YSQL and DocDB metadata for the child
partition tables following an ALTER, resulting in failure of future
DQL and DML operations.

Original Diff: https://phabricator.dev.yugabyte.com/D14685
Original Commit: ff176a5

Conflict resolution:
Since the number of tests has become too high to fit into an acceptable
timeout, the tests were split into multiple tests before the original
commit was pushed into master. However commit
ccabdf7 that introduces this
split cannot be back ported to 2.6 as it requires other commits before it.

Adding the tests introduced by this commit further increases the possibility
of the test failing due to timeout. Hence in this commit, introduce a new
file TestYbPgPartitions and only run the tests introduced by YB into its
schedule. This simple split does not require any previous commits and will
also not run into timeout issues.
Starting 2.8 release onwards, the split tests are present, so there will not be
any divergence from 2.8 onwards.

Test Plan:
Jenkins: rebase: 2.6
ybd --scb --sj --java-test org.yb.pgsql.TestPgRegressPartitions
ybd --scb --sj --java-test org.yb.pgsql.TestYbPgPartitions

Reviewers: myang

Reviewed By: myang

Subscribers: yql

Differential Revision: https://phabricator.dev.yugabyte.com/D15823
deeps1991 added a commit that referenced this issue Mar 16, 2022
…able affects select on child

Summary:
Failing example:
```
-- Add column to parent partitioned table.
yugabyte@yugabyte=# alter table transactions add column foo varchar(30);
ALTER TABLE

-- Column is added to child partition in PG metadata.
yugabyte@yugabyte=# \d transactions_eu;
                        Table "public.transactions_eu"
    Column     |            Type             | Collation | Nullable | Default
---------------+-----------------------------+-----------+----------+---------
 user_id       | integer                     |           | not null |
 geo_partition | character varying           |           | not null |
 foo           | character varying(30)       |           |          |
Partition of: transactions FOR VALUES IN ('EU', 'EMEA')

-- Select data from child
localhost:5433 yugabyte@yugabyte=# select * from transactions_eu;
ERROR:  Invalid argument: Invalid column number 8
```

The above issue is happening because ALTER on parent partitioned table
recursively reflects the ALTER operation on the child partitions by
the PG layer. However, this does not happen on the DocDB side.
This leads to mismatch in the YSQL and DocDB metadata for the child
partition tables following an ALTER, resulting in failure of future
DQL and DML operations.

Original Diff: https://phabricator.dev.yugabyte.com/D14685
Original Commit: ff176a5

Test Plan:
Jenkins: rebase: 2.12
ybd --scb --sj --java-test org.yb.pgsql.TestPgRegressPartitions

Reviewers: myang

Reviewed By: myang

Subscribers: yql

Differential Revision: https://phabricator.dev.yugabyte.com/D15820
deeps1991 added a commit that referenced this issue Mar 28, 2022
…le affects select on child

Summary:
Failing example:
```
-- Add column to parent partitioned table.
yugabyte@yugabyte=# alter table transactions add column foo varchar(30);
ALTER TABLE

-- Column is added to child partition in PG metadata.
yugabyte@yugabyte=# \d transactions_eu;
                        Table "public.transactions_eu"
    Column     |            Type             | Collation | Nullable | Default
---------------+-----------------------------+-----------+----------+---------
 user_id       | integer                     |           | not null |
 geo_partition | character varying           |           | not null |
 foo           | character varying(30)       |           |          |
Partition of: transactions FOR VALUES IN ('EU', 'EMEA')

-- Select data from child
localhost:5433 yugabyte@yugabyte=# select * from transactions_eu;
ERROR:  Invalid argument: Invalid column number 8
```

The above issue is happening because ALTER on parent partitioned table
recursively reflects the ALTER operation on the child partitions by
the PG layer. However, this does not happen on the DocDB side.
This leads to mismatch in the YSQL and DocDB metadata for the child
partition tables following an ALTER, resulting in failure of future
DQL and DML operations.

Original Diff: https://phabricator.dev.yugabyte.com/D14685
Original Commit: ff176a5

Test Plan: ybd --scb --sj --java-test org.yb.pgsql.TestPgRegressPartitions

Reviewers: myang

Reviewed By: myang

Subscribers: dfelsing, yql

Differential Revision: https://phabricator.dev.yugabyte.com/D16021
@deeps1991 deeps1991 moved this to Done in YQL-beta Mar 28, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/ysql Yugabyte SQL (YSQL)
Projects
Status: Done
Development

No branches or pull requests

4 participants