Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CH] Invalid number of rows in Chunk column Nullable(Int64) position 7: expected 1, got 0: While executing GraceMergingAggregatedTransform #6878

Closed
lgbo-ustc opened this issue Aug 16, 2024 · 2 comments · Fixed by #6886
Closed

[CH] Invalid number of rows in Chunk column Nullable(Int64) position 7: expected 1, got 0: While executing GraceMergingAggregatedTransform #6878

lgbo-ustc opened this issue Aug 16, 2024 · 2 comments · Fixed by #6886
Labels
bug Something isn't working triage

Comments

@lgbo-ustc
Copy link
Contributor

Backend

CH (ClickHouse)

Bug description

[Expected behavior] and [actual behavior].

Job aborted due to stage failure: Task 13 in stage 107.0 failed 4 times, most recent failure: Lost task 13.3 in stage 107.0 (TID 15552) (sg-dn3252.bigdata.bigo.inner executor 1546): org.apache.gluten.exception.GlutenException: org.apache.gluten.exception.GlutenException: org.apache.gluten.exception.GlutenException: Invalid number of rows in Chunk column Nullable(Int64) position 7: expected 1, got 0: While executing GraceMergingAggregatedTransform
0. Poco::Exception::Exception(String const&, int) @ 0x0000000012be2059
1. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000636dfdb
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000006248c6c
3. DB::Exception::Exception<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x000000000644058b
4. DB::Chunk::checkNumRowsIsConsistent() @ 0x000000001013b1f7
5. DB::Chunk::Chunk(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long) @ 0x000000001013b028
6. local_engine::GraceMergingAggregatedTransform::work() @ 0x0000000006882f39
7. DB::ExecutionThreadContext::executeTask() @ 0x000000001015ea22
8. DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000010154930
9. DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x00000000101544c9
10. DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x0000000010165274
11. DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x00000000101653d9
12. local_engine::LocalExecutor::hasNext() @ 0x00000000067466c9
13. Java_org_apache_gluten_vectorized_BatchIterator_nativeHasNext @ 0x00000000062339d7

Spark version

None

Spark configurations

No response

System information

No response

Relevant logs

No response
@lgbo-ustc lgbo-ustc added bug Something isn't working triage labels Aug 16, 2024
@lgbo-ustc
Copy link
Contributor Author

lgbo-ustc commented Aug 16, 2024 •
edited
Loading

There are two columns with the same name (sum(col_3))

      Positions:
        GraceMergingAggregatedStep
        Header: right_1.col_1 Nullable(Int64)
                col_4 Nullable(Int64)
                col_2 Nullable(Int64)
                col_1 Nullable(String)
                min(right_1.col_0) String
                sum(col_3) Nullable(Int64)
                sum(right_2.col_2) Nullable(Decimal(38, 6))
                sum(col_3) Nullable(Int64)
                sum(checkDecimalOverflowSparkOrNull(minus(CAST(CAST(right_2.col_2,Nullable(D_3),Nullable(Decimal(39, 6))_5),CAST(col_3,Nullable(D_4)),38_6,6_7)) Nullable(Decimal(38, 6))
        Keys: right_1.col_1, col_4, col_2, col_1, min(right_1.col_0), sum(col_3)
        Aggregates:
            sum(right_2.col_2)
              Function: sum(Nullable(Decimal(37, 6))) → Nullable(Decimal(38, 6))
              Arguments: right_2.col_2
            sum(col_3)
              Function: sum(Nullable(Int64)) → Nullable(Int64)
              Arguments: col_3
            sum(checkDecimalOverflowSparkOrNull(minus(CAST(CAST(right_2.col_2,Nullable(D_3),Nullable(Decimal(39, 6))_5),CAST(col_3,Nullable(D_4)),38_6,6_7))
              Function: sum(Nullable(Decimal(38, 6))) → Nullable(Decimal(38, 6))
              Arguments: checkDecimalOverflowSparkOrNull(minus(CAST(CAST(right_2.col_2,Nullable(D_3),Nullable(Decimal(39, 6))_5),CAST(col_3,Nullable(D_4)),38_6,6_7)
          Expression (Project)
          Header: right_1.col_1 Nullable(Int64)
                  col_4 Nullable(Int64)
                  col_2 Nullable(Int64)
                  col_1 Nullable(String)
                  min(right_1.col_0) String
                  sum(col_3) Nullable(Int64)
                  right_2.col_2 Nullable(Decimal(37, 6))
                  col_3 Nullable(Int64)
                  checkDecimalOverflowSparkOrNull(minus(CAST(CAST(right_2.col_2,Nullable(D_3),Nullable(Decimal(39, 6))_5),CAST(col_3,Nullable(D_4)),38_6,6_7) Nullable(Decimal(38, 6))

The new sum(col_3) is empty.

right_1.col_1 Nullable(Int64) Nullable(size = 1, Int64(size = 1), UInt8(size = 1)),
col_4 Nullable(Int64) Nullable(size = 1, Int64(size = 1), UInt8(size = 1)),
col_2 Nullable(Int64) Nullable(size = 1, Int64(size = 1), UInt8(size = 1)),
col_1 Nullable(String) Nullable(size = 1, String(size = 1), UInt8(size = 1)),
min(right_1.col_0) String String(size = 1),
sum(col_3) Nullable(Int64) Nullable(size = 1, Int64(size = 1), UInt8(size = 1)),
sum(right_2.col_2) Nullable(Decimal(38, 6)) Nullable(size = 1, Decimal128(size = 1), UInt8(size = 1)),
sum(col_3) Nullable(Int64) Nullable(size = 0, Int64(size = 0), UInt8(size = 0)),
sum(checkDecimalOverflowSparkOrNull(minus(CAST(CAST(right_2.col_2,Nullable(D_3),Nullable(Decimal(39, 6))_5),CAST(col_3,Nullable(D_4)),38_6,6_7)) Nullable(Decimal(38, 6)) Nullable(size = 1, Decimal128(size = 1), UInt8(size = 1))

Following is the whole plan segement

Expression (Project for output schema)
Header: refund_uid#1077 Nullable(String)
        refund_diamonds#1078 Nullable(Int64)
        ruid#1079 Nullable(String)
        countrycode#1080 Nullable(String)
        min_date#1081 Nullable(String)
        rl_diamonds#1082 Nullable(Int64)
        total_cost#1083 Nullable(Int64)
        total_win#1084 Nullable(Int64)
        net_cost#1085 Nullable(Int64)
        day#1086 String
Positions:
  Expression (Rename Output)
  Header: refund_uid#1077 Nullable(String)
          refund_diamonds#1078 Nullable(Int64)
          ruid#1079 Nullable(String)
          countrycode#1080 Nullable(String)
          min_date#1081 String
          rl_diamonds#1082 Nullable(Int64)
          total_cost#1083 Nullable(Int64)
          total_win#1084 Nullable(Int64)
          net_cost#1085 Nullable(Int64)
          day#1086 String
  Positions:
    Expression (Project)
    Header: CAST(right_1.col_1,Nullable(S_12) Nullable(String)
            col_4 Nullable(Int64)
            CAST(col_2,Nullable(S_13) Nullable(String)
            col_1 Nullable(String)
            min(right_1.col_0) String
            sum(col_3) Nullable(Int64)
            CAST(roundHalfUp(sum(right_2.col_2),0_14),Nullable(I_15) Nullable(Int64)
            roundHalfUp(sum(col_3),0_16) Nullable(Int64)
            CAST(roundHalfUp(sum(checkDecimalOverflowSparkOrNull(minus(CAST(CAST(right_2.col_2,Nullable(D_3),Nullable(Decimal(39, 6))_5),CAST(col_3,Nullable(D_4)),38_6,6_7)),0_17),Nullable(I_18) Nullable(Int64)
            2024-08-15_19 String
    Positions:
      Expression (Post-projection for aggregate)
      Header: right_1.col_1 Nullable(Int64)
              col_4 Nullable(Int64)
              col_2 Nullable(Int64)
              col_1 Nullable(String)
              min(right_1.col_0) String
              sum(col_3) Nullable(Int64)
              sum(right_2.col_2) Nullable(Decimal(38, 6))
              sum(col_3) Nullable(Int64)
              sum(checkDecimalOverflowSparkOrNull(minus(CAST(CAST(right_2.col_2,Nullable(D_3),Nullable(Decimal(39, 6))_5),CAST(col_3,Nullable(D_4)),38_6,6_7)) Nullable(Decimal(38, 6))
      Positions:
        GraceMergingAggregatedStep
        Header: right_1.col_1 Nullable(Int64)
                col_4 Nullable(Int64)
                col_2 Nullable(Int64)
                col_1 Nullable(String)
                min(right_1.col_0) String
                sum(col_3) Nullable(Int64)
                sum(right_2.col_2) Nullable(Decimal(38, 6))
                sum(col_3) Nullable(Int64)
                sum(checkDecimalOverflowSparkOrNull(minus(CAST(CAST(right_2.col_2,Nullable(D_3),Nullable(Decimal(39, 6))_5),CAST(col_3,Nullable(D_4)),38_6,6_7)) Nullable(Decimal(38, 6))
        Keys: right_1.col_1, col_4, col_2, col_1, min(right_1.col_0), sum(col_3)
        Aggregates:
            sum(right_2.col_2)
              Function: sum(Nullable(Decimal(37, 6))) → Nullable(Decimal(38, 6))
              Arguments: right_2.col_2
            sum(col_3)
              Function: sum(Nullable(Int64)) → Nullable(Int64)
              Arguments: col_3
            sum(checkDecimalOverflowSparkOrNull(minus(CAST(CAST(right_2.col_2,Nullable(D_3),Nullable(Decimal(39, 6))_5),CAST(col_3,Nullable(D_4)),38_6,6_7))
              Function: sum(Nullable(Decimal(38, 6))) → Nullable(Decimal(38, 6))
              Arguments: checkDecimalOverflowSparkOrNull(minus(CAST(CAST(right_2.col_2,Nullable(D_3),Nullable(Decimal(39, 6))_5),CAST(col_3,Nullable(D_4)),38_6,6_7)
          Expression (Project)
          Header: right_1.col_1 Nullable(Int64)
                  col_4 Nullable(Int64)
                  col_2 Nullable(Int64)
                  col_1 Nullable(String)
                  min(right_1.col_0) String
                  sum(col_3) Nullable(Int64)
                  right_2.col_2 Nullable(Decimal(37, 6))
                  col_3 Nullable(Int64)
                  checkDecimalOverflowSparkOrNull(minus(CAST(CAST(right_2.col_2,Nullable(D_3),Nullable(Decimal(39, 6))_5),CAST(col_3,Nullable(D_4)),38_6,6_7) Nullable(Decimal(38, 6))
          Positions:
            Expression (Project)
            Header: right_1.col_1 Nullable(Int64)
                    col_4 Nullable(Int64)
                    col_2 Nullable(Int64)
                    col_1 Nullable(String)
                    min(right_1.col_0) String
                    sum(col_3) Nullable(Int64)
                    col_0 Nullable(Int64)
                    right_2.col_1 Nullable(String)
                    right_2.col_2 Nullable(Decimal(37, 6))
                    col_3 Nullable(Int64)
            Positions:
              Expression (Reorder Join Output)
              Header: right_1.col_1 Nullable(Int64)
                      col_4 Nullable(Int64)
                      col_2 Nullable(Int64)
                      col_1 Nullable(String)
                      min(right_1.col_0) String
                      sum(col_3) Nullable(Int64)
                      col_0 Nullable(Int64)
                      right_2.col_1 Nullable(String)
                      right_2.col_2 Nullable(Decimal(37, 6))
                      col_3 Nullable(Int64)
              Positions:
                Join (HASH_JOIN)
                Header: right_1.col_1 Nullable(Int64)
                        col_4 Nullable(Int64)
                        col_2 Nullable(Int64)
                        col_1 Nullable(String)
                        min(right_1.col_0) String
                        sum(col_3) Nullable(Int64)
                        right_2.col_1 Nullable(String)
                        right_2.col_2 Nullable(Decimal(37, 6))
                        col_3 Nullable(Int64)
                        col_0 Nullable(Int64)
                Type: INNER
                Strictness: ALL
                Algorithm: GraceHashJoin
                Clauses: [(col_2) = (col_0)]
                  Expression (Remove nullable properties)
                  Header: right_1.col_1 Nullable(Int64)
                          col_4 Nullable(Int64)
                          col_2 Nullable(Int64)
                          col_1 Nullable(String)
                          min(right_1.col_0) String
                          sum(col_3) Nullable(Int64)
                  Positions:
                    Filter (WHERE)
                    Header: right_1.col_1 Nullable(Int64)
                            col_4 Nullable(Int64)
                            col_2 Nullable(Int64)
                            col_1 Nullable(String)
                            min(right_1.col_0) Nullable(String)
                            sum(col_3) Nullable(Int64)
                    Filter column: isNotNull(min(right_1.col_0)) (removed)
                    Positions:
                      GraceMergingAggregatedStep
                      Header: right_1.col_1 Nullable(Int64)
                              col_4 Nullable(Int64)
                              col_2 Nullable(Int64)
                              col_1 Nullable(String)
                              min(right_1.col_0) Nullable(String)
                              sum(col_3) Nullable(Int64)
                      Keys: right_1.col_1, col_4, col_2, col_1
                      Aggregates:
                          min(right_1.col_0)
                            Function: min(Nullable(String)) → Nullable(String)
                            Arguments: right_1.col_0
                          sum(col_3)
                            Function: sum(Nullable(Int64)) → Nullable(Int64)
                            Arguments: col_3
                        Expression (Project)
                        Header: right_1.col_0 Nullable(String)
                                right_1.col_1 Nullable(Int64)
                                col_2 Nullable(Int64)
                                col_3 Nullable(Int64)
                                col_4 Nullable(Int64)
                                col_1 Nullable(String)
                        Positions:
                          Expression (Project)
                          Header: col_0 String
                                  col_1 Nullable(String)
                                  right_1.col_0 Nullable(String)
                                  right_1.col_1 Nullable(Int64)
                                  col_2 Nullable(Int64)
                                  col_3 Nullable(Int64)
                                  col_4 Nullable(Int64)
                          Positions:
                            Expression (Reorder Join Output)
                            Header: col_0 String
                                    col_1 Nullable(String)
                                    CAST(trim(col_0),Nullable(I_0) Nullable(Int64)
                                    right_1.col_0 Nullable(String)
                                    right_1.col_1 Nullable(Int64)
                                    col_2 Nullable(Int64)
                                    col_3 Nullable(Int64)
                                    col_4 Nullable(Int64)
                            Positions:
                              Join (HASH_JOIN)
                              Header: col_0 String
                                      col_1 Nullable(String)
                                      CAST(trim(col_0),Nullable(I_0) Nullable(Int64)
                                      right_1.col_0 Nullable(String)
                                      right_1.col_1 Nullable(Int64)
                                      col_3 Nullable(Int64)
                                      col_4 Nullable(Int64)
                                      col_2 Nullable(Int64)
                              Type: INNER
                              Strictness: ALL
                              Algorithm: GraceHashJoin
                              Clauses: [(CAST(trim(col_0),Nullable(I_0)) = (col_2)]
                                Expression (Project)
                                Header: col_0 String
                                        col_1 Nullable(String)
                                        CAST(trim(col_0),Nullable(I_0) Nullable(Int64)
                                Positions:
                                  ReadFromPreparedSource (Read From Java Iter)
                                  Header: col_0 String
                                          col_1 Nullable(String)
                                Expression (Right Table Rename)
                                Header: right_1.col_0 Nullable(String)
                                        right_1.col_1 Nullable(Int64)
                                        col_2 Nullable(Int64)
                                        col_3 Nullable(Int64)
                                        col_4 Nullable(Int64)
                                Positions:
                                  ReadFromPreparedSource (Read From Java Iter)
                                  Header: col_0 Nullable(String)
                                          col_1 Nullable(Int64)
                                          col_2 Nullable(Int64)
                                          col_3 Nullable(Int64)
                                          col_4 Nullable(Int64)
                  Expression (Right Table Rename)
                  Header: col_0 Nullable(Int64)
                          right_2.col_1 Nullable(String)
                          right_2.col_2 Nullable(Decimal(37, 6))
                          col_3 Nullable(Int64)
                  Positions:
                    ReadFromPreparedSource (Read From Java Iter)
                    Header: col_0 Nullable(Int64)
                            col_1 Nullable(String)
                            col_2 Nullable(Decimal(37, 6))
                            col_3 Nullable(Int64)

@lgbo-ustc lgbo-ustc mentioned this issue Aug 16, 2024
Merged
@lgbo-ustc
Copy link
Contributor Author

lgbo-ustc commented Aug 16, 2024 •
edited
Loading

May be related to #4668, It's the case that two stage aggregation without shuffle, the two stages are merged into one stage.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working triage
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[GLUTEN-6878][CH] Avoid name collisions in naming aggregate result bigo-sg/gluten
1 participant
@lgbo-ustc