Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Wrong LIMIT rewrite while GROUP BY and ORDER BY on different items #2062

Closed
joooohnli opened this issue Mar 20, 2019 · 10 comments
Closed

Wrong LIMIT rewrite while GROUP BY and ORDER BY on different items #2062

joooohnli opened this issue Mar 20, 2019 · 10 comments
Assignees
Milestone

Comments

@joooohnli
Copy link

Bug Report

For English only, other languages will not accept.

Before report a bug, make sure you have:

Please pay attention on issues you submitted, because we maybe need more details.
If no response more than 7 days and we cannot reproduce it on current information, we will close it.

Please answer these questions before submitting your issue. Thanks!

Which version of ShardingSphere did you use?

3.1.0

Which project did you use? Sharding-JDBC or Sharding-Proxy?

Sharding-JDBC

Expected behavior

limit 0, 10

Actual behavior

 SQL: select max(sid) as sid, max(sendTime) as sendTime, objectId, sum(0) as unreadCount from TEST where memberId = ? and status != -1 group by objectId order by sendTime desc limit ?, ?
SQL-BindValue: 52146792, 0, 2147483647

Reason analyze (If you can)

logic error:

io.shardingsphere.core.routing.router.sharding.ParsingSQLRouter#processLimit
image

Steps to reproduce the behavior, such as: SQL to execute, sharding rule configuration, when exception occur etc.

Run it with PreparedStatement:

select max(sid) as sid, max(sendTime) as sendTime, objectId, sum(0) as unreadCount from TEST where memberId = ? and status != -1 group by objectId order by sendTime desc limit ?, ?

Example codes for reproduce this issue (such as a github link).

@tuohai666
Copy link
Member

@joooohnli Please try to describe the "Expected behavior" and "Actual behavior" clearly. I'm not saying you must use many words. Think about it very carefully before you responsd, thanks.

@joooohnli
Copy link
Author

@tuohai666

Expected behavior

What I input is limit 0, 10

Actual behavior

What I really got is limit 0, 2147483647 you can see as below:

SQL: select max(sid) as sid, max(sendTime) as sendTime, objectId, sum(0) as unreadCount from TEST where memberId = ? and status != -1 group by objectId order by sendTime desc limit ?, ?
SQL-BindValue: 52146792, 0, 2147483647

@tuohai666
Copy link
Member

I am not very sure what's the problem. I don't know where the log come from. So I must know the behavior.
Let me guess what the program behavior you haven seen:

  1. You input a sql and expect a result with 10 rows. If you don't use Sharding-JDBC, you connect to mysql directly, execute the same sql, you got 10 rows. That means your expect is right.
  2. But you got and saw 2147483647 rows in your screen.

Please answer are the two above right? If they are right, please prove you got 2147483647 rows, you don't need to past all 2147483647, but try to prove it. Don't use log to prove it, I think the log is not from Sharding-JDBC.

Also, please provide the example codes and configurations.

BTW, What I input is limit 0, 10 is your behavior, it's not the program behavior.

@joooohnli
Copy link
Author

@tuohai666
Here is the example code

            connection = dataSource.getConnection();
            PreparedStatement preparedStatement = connection.prepareStatement("select max(sid) as sid, max(sendTime) as sendTime, objectId, sum(0) as unreadCount from TEST where memberId = ? and status != -1 group by objectId order by sendTime desc limit ?, ?");
            preparedStatement.setInt(1, 123);
            preparedStatement.setInt(2, 0);
            preparedStatement.setInt(3, 10);
            ResultSet resultSet = preparedStatement.executeQuery();

And TEST is just a single table in defaultDataSource. (in fact this issue also occurs to sharding table)

The log comes from my jdbc monitor, it shows the real sql that jdbc sends to physical db.

If you think the log is not convincing. I can show you the debug info:
io.shardingsphere.core.parsing.parser.context.limit.Limit#rewrite
image

Actually you could have reproduced this bug according to the Steps to reproduce the behavior.

@tuohai666
Copy link
Member

  1. Set sql.show : true to get a Sharding-JDBC log.
  2. I have got your monitor log and understand very clearly, but we talk only Sharding-JDBC log from now on. Don't put your debug info here, thanks.
  3. Your monitor log is never a behavior of Sharding-JDBC. To describe a behavior show the result rows please. I will not ask you to put 2147483647 rows here, you can only put 11 rows to prove there is a bug.
  4. I can't reproduce it because I don't have the configuration, I even don't know what's in your database.
  5. Focus on the result rows. It's not helpful to analyze before you find out what you want and what you acutally get.
  6. More detail. Please answer these two questions:
    Access MySQL Server without Sharding-JDBC: how many rows will return when you execute this SQL?
    Access MySQL Server with Sharding-JDBC: how many rows will return when you execute this SQL?
    And put the results here.

@joooohnli
Copy link
Author

2019-03-21 14:57:16.416  INFO 70268 --- [nio-8090-exec-1] ShardingSphere-SQL                       : Rule Type: sharding
2019-03-21 14:57:16.416  INFO 70268 --- [nio-8090-exec-1] ShardingSphere-SQL                       : Logic SQL: select max(sid) as sid, max(sendTime) as sendTime, objectId, sum(0) as unreadCount from TEST where memberId = ? and status != -1 group by objectId order by sendTime desc limit ?, ?
2019-03-21 14:57:16.416  INFO 70268 --- [nio-8090-exec-1] ShardingSphere-SQL                       : SQLStatement: SelectStatement(super=DQLStatement(super=io.shardingsphere.core.parsing.parser.sql.dql.select.SelectStatement@4db4cf4f), containStar=false, firstSelectItemStartPosition=7, selectListLastPosition=83, groupByLastPosition=153, items=[AggregationSelectItem(type=MAX, innerExpression=(sid), alias=Optional.of(sid), derivedAggregationSelectItems=[], index=-1), AggregationSelectItem(type=MAX, innerExpression=(sendTime), alias=Optional.of(sendTime), derivedAggregationSelectItems=[], index=-1), CommonSelectItem(expression=objectId, alias=Optional.absent()), AggregationSelectItem(type=SUM, innerExpression=(0), alias=Optional.of(unreadCount), derivedAggregationSelectItems=[], index=-1)], groupByItems=[OrderItem(owner=Optional.absent(), name=Optional.of(objectId), orderDirection=ASC, nullOrderDirection=ASC, index=-1, expression=null, alias=Optional.absent())], orderByItems=[OrderItem(owner=Optional.absent(), name=Optional.of(sendTime), orderDirection=DESC, nullOrderDirection=ASC, index=-1, expression=null, alias=Optional.of(sendTime))], limit=Limit(offset=LimitValue(value=0, index=1, boundOpened=false), rowCount=LimitValue(value=10, index=2, boundOpened=false)), subQueryStatement=null, subQueryStatements=[], subQueryConditions=[])
2019-03-21 14:57:16.416  INFO 70268 --- [nio-8090-exec-1] ShardingSphere-SQL                       : Actual SQL: test ::: select max(sid) as sid, max(sendTime) as sendTime, objectId, sum(0) as unreadCount from TEST where memberId = ? and status != -1 group by objectId order by sendTime desc limit ?, ? ::: [[123, 0, 2147483647]]
  1. Sharding log as ablove.
  2. Are you sure the debug info does not help. Do you think rewriting limit count to 2147483647 is reseassonable in this case?
  3. ok
    image
    image
  4. There is nothing about my database. Sharding-jdbc rewrites the sql, it happens before the sql being sent to the database.
  5. I am much clear what I want and what I get, which was represented in the issue report.
  6. a. 10 rows; b. more than 10 rows

This issue happened on my prod environment and leaded to some horrible consequences. I could 100% reproduce it. There is no special precondition. You can also reproduce it according to the Steps to reproduce the behavior. Just try it.

@tuohai666
Copy link
Member

Thanks for your effort. This is the exactly information I wanted.
I'll find out whether this problem is a known issue.
If it is a new bug, it will be fixed in release 4.0.0-RC2.

tuohai666 pushed a commit that referenced this issue Mar 26, 2019
#2062, fix SQL rewrite issue (#2086)
terrymanu added a commit that referenced this issue Mar 26, 2019
#2062, Wrong LIMIT rewrite while GROUP BY and ORDER BY on different items
@tuohai666
Copy link
Member

@joooohnli Resolved. Thanks for your report. You can try with dev branch now. This modification will be included in 4.0.0-RC1 which will be released this week.

@Anricx
Copy link

Anricx commented Apr 5, 2019

so, will there be no hotfix for 3.x? @terrymanu

@lyflyy
Copy link

lyflyy commented Jun 14, 2021

I also had this issue in 【4.1.1】

Bug Report

For English only, other languages will not accept.

Before report a bug, make sure you have:

Please pay attention on issues you submitted, because we maybe need more details.
If no response anymore and we cannot reproduce it on current information, we will close it.

Please answer these questions before submitting your issue. Thanks!

Which version of ShardingSphere did you use?

4.1.1

Which project did you use? ShardingSphere-JDBC or ShardingSphere-Proxy?

ShardingSphere-JDBC

Expected behavior

I want to return 10 pieces of data in the SQL result

Actual behavior

The rewritten SQL queries 2147483647 pieces of data

Reason analyze (If you can)

Steps to reproduce the behavior, such as: SQL to execute, sharding rule configuration, when exception occur etc.

ShardingSphere-SQL LOG detail

2021-06-14 16:40:46.205 DEBUG 12448 --- [nio-8080-exec-1] c.s.d.mapper.OrdersMapper.officialDemo   : ==>  Preparing: SELECT user_id, SUM(price) `sum` FROM orders where create_time > ? and create_time < ? GROUP BY user_id ORDER BY `sum` DESC LIMIT 0, 10 
2021-06-14 16:40:46.231 DEBUG 12448 --- [nio-8080-exec-1] c.s.d.mapper.OrdersMapper.officialDemo   : ==> Parameters: 2020-07-09 13:54:57.0(Timestamp), 2022-07-19 13:54:57.0(Timestamp)
2021-06-14 16:40:47.088  INFO 12448 --- [nio-8080-exec-1] ShardingSphere-SQL                       : Logic SQL: SELECT user_id, SUM(price) `sum` FROM orders
         
            where create_time > ? and create_time < ?
         
        GROUP BY user_id ORDER BY `sum` DESC LIMIT 0, 10
2021-06-14 16:40:47.088  INFO 12448 --- [nio-8080-exec-1] ShardingSphere-SQL                       : SQLStatement: SelectStatementContext(super=CommonSQLStatementContext(sqlStatement=org.apache.shardingsphere.sql.parser.sql.statement.dml.SelectStatement@109c4794, tablesContext=org.apache.shardingsphere.sql.parser.binder.segment.table.TablesContext@66e38b62), tablesContext=org.apache.shardingsphere.sql.parser.binder.segment.table.TablesContext@66e38b62, projectionsContext=ProjectionsContext(startIndex=7, stopIndex=31, distinctRow=false, projections=[ColumnProjection(owner=null, name=user_id, alias=Optional.empty), AggregationProjection(type=SUM, innerExpression=(price), alias=Optional[sum], derivedAggregationProjections=[], index=-1)]), groupByContext=org.apache.shardingsphere.sql.parser.binder.segment.select.groupby.GroupByContext@1bba7f5d, orderByContext=org.apache.shardingsphere.sql.parser.binder.segment.select.orderby.OrderByContext@3e02c417, paginationContext=org.apache.shardingsphere.sql.parser.binder.segment.select.pagination.PaginationContext@4c5084a4, containsSubquery=false)
2021-06-14 16:40:47.088  INFO 12448 --- [nio-8080-exec-1] ShardingSphere-SQL                       : Actual SQL: dbs0 ::: SELECT user_id, SUM(price) `sum` FROM orders_6
         
            where create_time > ? and create_time < ?
         
        GROUP BY user_id ORDER BY `sum` DESC LIMIT 0, 2147483647 ::: [2020-07-09 13:54:57.0, 2022-07-19 13:54:57.0]
2021-06-14 16:40:47.088  INFO 12448 --- [nio-8080-exec-1] ShardingSphere-SQL                       : Actual SQL: dbs1 ::: SELECT user_id, SUM(price) `sum` FROM orders_6
         
            where create_time > ? and create_time < ?
         
        GROUP BY user_id ORDER BY `sum` DESC LIMIT 0, 2147483647 ::: [2020-07-09 13:54:57.0, 2022-07-19 13:54:57.0]
2021-06-14 16:40:47.088  INFO 12448 --- [nio-8080-exec-1] ShardingSphere-SQL                       : Actual SQL: dbs2 ::: SELECT user_id, SUM(price) `sum` FROM orders_6
         
            where create_time > ? and create_time < ?
         
        GROUP BY user_id ORDER BY `sum` DESC LIMIT 0, 2147483647 ::: [2020-07-09 13:54:57.0, 2022-07-19 13:54:57.0]
2021-06-14 16:40:48.780 DEBUG 12448 --- [nio-8080-exec-1] c.s.d.mapper.OrdersMapper.officialDemo   : <==      Total: 10

Example codes for reproduce this issue (such as a github link).

  1. I use read/write separation and my configuration file is as follows
# 数据源 db0,db1
logging.level.com.sharding.demo.mapper=debug

spring.shardingsphere.props.sql.show=true
spring.shardingsphere.props.max.connections.size.per.query=1

mybatis-plus.mapper-locations=classpath:/mapper/*.xml

spring.shardingsphere.datasource.names = db0,db1,db2,dbs0,dbs1,dbs2

spring.shardingsphere.datasource.db0.type = com.zaxxer.hikari.HikariDataSource
spring.shardingsphere.datasource.db0.driver-class-name = com.mysql.cj.jdbc.Driver
spring.shardingsphere.datasource.db0.jdbc-url = jdbc:mysql://127.0.0.1:3306/db0?characterEncoding=utf8&useSSL=false
spring.shardingsphere.datasource.db0.username = root
spring.shardingsphere.datasource.db0.password = root

spring.shardingsphere.datasource.db1.type = com.zaxxer.hikari.HikariDataSource
spring.shardingsphere.datasource.db1.driver-class-name = com.mysql.cj.jdbc.Driver
spring.shardingsphere.datasource.db1.jdbc-url = jdbc:mysql://127.0.0.1:3306/db1?characterEncoding=utf8&useSSL=false
spring.shardingsphere.datasource.db1.username = root
spring.shardingsphere.datasource.db1.password = root

spring.shardingsphere.datasource.db2.type = com.zaxxer.hikari.HikariDataSource
spring.shardingsphere.datasource.db2.driver-class-name = com.mysql.cj.jdbc.Driver
spring.shardingsphere.datasource.db2.jdbc-url = jdbc:mysql://127.0.0.1:3306/db2?characterEncoding=utf8&useSSL=false
spring.shardingsphere.datasource.db2.username = root
spring.shardingsphere.datasource.db2.password = root

spring.shardingsphere.datasource.dbs0.type = com.zaxxer.hikari.HikariDataSource
spring.shardingsphere.datasource.dbs0.driver-class-name = com.mysql.cj.jdbc.Driver
spring.shardingsphere.datasource.dbs0.jdbc-url = jdbc:mysql://127.0.0.1:3307/db0?characterEncoding=utf8&useSSL=false
spring.shardingsphere.datasource.dbs0.username = root
spring.shardingsphere.datasource.dbs0.password = root

spring.shardingsphere.datasource.dbs1.type = com.zaxxer.hikari.HikariDataSource
spring.shardingsphere.datasource.dbs1.driver-class-name = com.mysql.cj.jdbc.Driver
spring.shardingsphere.datasource.dbs1.jdbc-url = jdbc:mysql://127.0.0.1:3307/db1?characterEncoding=utf8&useSSL=false
spring.shardingsphere.datasource.dbs1.username = root
spring.shardingsphere.datasource.dbs1.password = root

spring.shardingsphere.datasource.dbs2.type = com.zaxxer.hikari.HikariDataSource
spring.shardingsphere.datasource.dbs2.driver-class-name = com.mysql.cj.jdbc.Driver
spring.shardingsphere.datasource.dbs2.jdbc-url = jdbc:mysql://127.0.0.1:3307/db2?characterEncoding=utf8&useSSL=false
spring.shardingsphere.datasource.dbs2.username = root
spring.shardingsphere.datasource.dbs2.password = root

#master-slave   基于master1和master2主从集群实现读写分离
spring.shardingsphere.sharding.master-slave-rules.db0.master-data-source-name=db0
spring.shardingsphere.sharding.master-slave-rules.db0.slave-data-source-names=dbs0

spring.shardingsphere.sharding.master-slave-rules.db1.master-data-source-name=db1
spring.shardingsphere.sharding.master-slave-rules.db1.slave-data-source-names=dbs1

spring.shardingsphere.sharding.master-slave-rules.db2.master-data-source-name=db2
spring.shardingsphere.sharding.master-slave-rules.db2.slave-data-source-names=dbs2

#多个从库的时候使用负载均衡
spring.shardingsphere.masterslave.load-balance-algorithm-type=ROUND_ROBIN

# user 表策略 用户按照性别进行分库, 2个库, id取模进行分表 3张表, db_male ,db_female user_0, user_1, user_2
# 分库策略 根据id取模确定数据进哪个数据库
spring.shardingsphere.sharding.tables.user.database-strategy.inline.sharding-column = sex
spring.shardingsphere.sharding.tables.user.database-strategy.inline.algorithm-expression = db$->{sex % 2}
# 分表策略
spring.shardingsphere.sharding.tables.user.actual-data-nodes = db$->{0..1}.user_$->{0..2}
# 分表字段member_id
spring.shardingsphere.sharding.tables.user.table-strategy.inline.sharding-column = id
# 分表策略 根据member_id取模,确定数据最终落在那个表中
spring.shardingsphere.sharding.tables.user.table-strategy.inline.algorithm-expression = user_$->{id % 3}
# 使用SNOWFLAKE算法生成主键
spring.shardingsphere.sharding.tables.user.key-generator.column = id
spring.shardingsphere.sharding.tables.user.key-generator.type = SNOWFLAKE


# 商品按照类型取模分库,3个库,id取模进行分表 db_type1, db_type2, db_type3, goods_0, goods_1, goods_2
# 分库策略 根据id取模确定数据进哪个数据库
spring.shardingsphere.sharding.tables.goods.database-strategy.inline.sharding-column = type
spring.shardingsphere.sharding.tables.goods.database-strategy.inline.algorithm-expression = db$->{type % 3}
# 分表策略
spring.shardingsphere.sharding.tables.goods.actual-data-nodes = db$->{0..2}.goods_$->{0..2}
# 分表字段member_id
spring.shardingsphere.sharding.tables.goods.table-strategy.inline.sharding-column = id
# 分表策略 根据member_id取模,确定数据最终落在那个表中
spring.shardingsphere.sharding.tables.goods.table-strategy.inline.algorithm-expression = goods_$->{id % 3}
# 使用SNOWFLAKE算法生成主键
spring.shardingsphere.sharding.tables.goods.key-generator.column = id
spring.shardingsphere.sharding.tables.goods.key-generator.type = SNOWFLAKE


# 订单按照下单月份进行分表,年份进行分库, db_2020,db_2021,db_2022
#声明虚拟表
spring.shardingsphere.sharding.tables.orders.actual-data-nodes=db$->{0..2}.orders_$->{0..11}
#声明表内的主键
spring.shardingsphere.sharding.tables.orders.key-generator.column=id
#声明主键生成策略
spring.shardingsphere.sharding.tables.orders.key-generator.type=SNOWFLAKE
##声明根据哪个字段进行分片
spring.shardingsphere.sharding.tables.orders.database-strategy.standard.sharding-column=create_time
##自定义分片规则类
## 分库规则orders_auto_increment
spring.shardingsphere.sharding.tables.orders.database-strategy.standard.precise-algorithm-class-name=com.sharding.demo.algorithm.OrderDatabaseShardingAlgorithm
spring.shardingsphere.sharding.tables.orders.database-strategy.standard.range-algorithm-class-name=com.sharding.demo.algorithm.OrderDatabaseShardingRangeAlgorithm
#声明根据哪个字段进行分片
spring.shardingsphere.sharding.tables.orders.table-strategy.standard.sharding-column=create_time
#自定义分片规则类
spring.shardingsphere.sharding.tables.orders.table-strategy.standard.precise-algorithm-class-name=com.sharding.demo.algorithm.OrderTableShardingAlgorithm
spring.shardingsphere.sharding.tables.orders.table-strategy.standard.range-algorithm-class-name=com.sharding.demo.algorithm.OrderTableShardingRangeAlgorithm

# 配置公共表
# 一个实体类对应两张表,覆盖
spring.main.allow-bean-definition-overriding=true
spring.shardingsphere.sharding.broadcast-tables=common_dict
# 配置数据库中 t_dict 表主键 dict_id 生成策略 SNOWFLAKE 雪花算法
spring.shardingsphere.sharding.tables.common_dict.key-generator.column=dict_id
spring.shardingsphere.sharding.tables.common_dict.key-generator.type=SNOWFLAKE
  1. i use mybatis
<select id="officialDemo" resultType="com.sharding.demo.vo.OffocialDemoVo">
    SELECT user_id, SUM(price) `sum` FROM orders
    <if test="orders.createTimeStart != null">
        where create_time > #{orders.createTimeStart} and create_time &lt; #{orders.createTimeEnd}
    </if>
    GROUP BY user_id ORDER BY `sum` DESC LIMIT 0, 10
</select>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants