Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix parsing of row events for MySQL8 partitioned table #355

Merged
merged 3 commits into from
Aug 30, 2021
Merged

Fix parsing of row events for MySQL8 partitioned table #355

merged 3 commits into from
Aug 30, 2021

Conversation

dongwook-chan
Copy link
Collaborator

@dongwook-chan dongwook-chan commented Aug 22, 2021

Since MySQL 8.0.16, partition info has been added into extra_data in row event.

Current code is aware of extra_data but doesn't parse it accurately. The program reads extra_data_length / 8 bytes from packet instead of correct byte-length extra_data_length - 2. This causes the library to skip parsing of extra_data.

In some cases exceptions are raised, and in other cases parse result shows incorrect field values as in issue #354. (undefined behavior)

below are the modifications made:

  1. Correct byte-length of extra_data to conform to replication protocol
    extra_data_length / 8 -> extra_data_length - 2

  2. Parse extra_data according to extra_data layout to get partition id ( or ndb info)

        +----------+--------------------------------------+
        |type_code |        extra_row_ndb_info            |
        +--- ------+--------------------------------------+
        | NDB      |Len of ndb_info |Format |ndb_data     |
        | 1 byte   |1 byte          |1 byte |len - 2 byte |
        +----------+----------------+-------+-------------+

        In case of INSERT/DELETE
        +-----------+----------------+
        | type_code | partition_info |
        +-----------+----------------+
        |   PART    |  partition_id  |
        | (1 byte)  |     2 byte     |
        +-----------+----------------+

        In case of UPDATE
        +-----------+------------------------------------+
        | type_code |        partition_info              |
        +-----------+--------------+---------------------+
        |   PART    | partition_id | source_partition_id |
        | (1 byte)  |    2 byte    |       2 byte        |
        +-----------+--------------+---------------------+
  1. data type tests were added to test parsing of extra_data, and correct value of prtition_id

Since MySQL 8.0.16, partition info has been [added](mysql/mysql-server@e11a540) into extra_data in row event.

Current code is aware of extra_data but doesn't parse it accurately.
Its length is defined as 'extra_data_length / 8'.
However, the [maximum length of partition info](https://github.com/mysql/mysql-server/blob/beb865a960b9a8a16cf999c323e46c5b0c67f21f/libbinlogevents/include/rows_event.h#L772-L814) is 5.
Wrong definition leads program to read 0 (= 5/8) bytes of extra_data.

This causes any row event on partitioned table leads to packet parse failure (undefined behavior).
In some cases errors occur, and in other cases parse result shows  incorrect filed values as in [issue #354](#354 (comment)).

1. Fix byte-length of extra_data to conform to repl protocol
    extra_data_length / 8 -> extra_data_length - 2
    Refer to [MySQL Document](https://dev.mysql.com/doc/internals/en/rows-event.html) and [MySQL source code]((https://github.com/mysql/mysql-server/blob/beb865a960b9a8a16cf999c323e46c5b0c67f21f/libbinlogevents/src/rows_event.cpp#L415)).

Above fixes failure of packet parsing, but following modification is further required to get partition info.

2. Parse extra_data according to [partition info layout](https://github.com/mysql/mysql-server/blob/beb865a960b9a8a16cf999c323e46c5b0c67f21f/libbinlogevents/include/rows_event.h#L772-L814).
test_extra_data -> test_partition_id
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants