Invalid access: Can not convert empty value

[lilinghai]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

daily run window function and the failed sql is select i, i, row_number() over w from window_function_0 window w as (partition by i order by id, i) order by 1, 2, 3, the full error info is Error 1105: other error for mpp stream: From MPP<query:<query_ts:0, local_query_id:0, server_id:0, start_ts:438529181194125313>,task_id:1>: Poco::Exception. Code: 1000, e.code() = 0, e.displayText() = Invalid access: Can not convert empty value., e.what() = Invalid access.

2. What did you expect to see? (Required)

3. What did you see instead (Required)

4. What is your TiFlash version? (Required)

[gengliqi]

I encountered this panic too.

[2023/01/05 18:01:29.371 +08:00] [ERROR] [Exception.cpp:89] ["Poco::Exception. Code: 1000, e.code() = 0, e.displayText() = Invalid access: Can not convert empty value., e.what() = Invalid access"] [source="DB::RawCppPtr DB::PreHandleSnapshot(DB::EngineStoreServerWrap *, DB::BaseBuffView, uint64_t, DB::SSTViewVec, uint64_t, uint64_t)"] [thread_id=23]

[CalvinNeo]

It is curious that we saw the same some errors in SchemaSyncService.

This may be a arm64 problem since pocoproject/poco#3331, however, we are not tested in arm64.

[CalvinNeo]

The mpp error is due to the schema syncer error

[source="DB::RawCppPtr DB::PreHandleSnapshot(DB::EngineStoreServerWrap *, DB::BaseBuffView, uint64_t, DB::SSTViewVec, uint64_t, uint64_t)"] [thread_id=50]
[2023/01/14 00:58:01.071 +08:00] [INFO] [TiDBSchemaSyncer.h:113] ["Start to sync schemas. current version is: 34 and try to sync schema version to: 39"] [source=SchemaSyncer] [thread_id=51]
[2023/01/14 00:58:01.071 +08:00] [DEBUG] [TiDBSchemaSyncer.h:171] ["Try load schema diffs."] [source=SchemaSyncer] [thread_id=51]
[2023/01/14 00:58:01.071 +08:00] [ERROR] [Exception.cpp:89] ["Poco::Exception. Code: 1000, e.code() = 0, e.displayText() = Invalid access: Can not convert empty value., e.what() = Invalid access"] [source="DB::RawCppPtr DB::PreHandleSnapshot(DB::EngineStoreServerWrap *, DB::BaseBuffView, uint64_t, DB::SSTViewVec, uint64_t, uint64_t)"] [thread_id=51]

[CalvinNeo]

Add log and see what the invalid data is

[CalvinNeo]

Restart solved

[CalvinNeo]

failed to deserialize {\"version\":40,\"type\":31,\"schema_id\":69,\"table_id\":71,\"old_table_id\":0,\"old_schema_id\":0,\"affected_options\":null}"

[JaySon-Huang]

Does Poco::JSON not support parsing "affected_options": null?

[CalvinNeo]

No, the regenerate_schema_map is missing.

...
 old_table_id = obj->getValue<Int64>("old_table_id");
 old_schema_id = obj->getValue<Int64>("old_schema_id");
 regenerate_schema_map = obj->getValue<bool>("regenerate_schema_map");
...

[JaySon-Huang]

Got it. So it is introduced by #6563 which tries to get a field that is not always set in SchemaDiff.

[CalvinNeo]

Ok, so seem this is due to the TiDB version is too old in the test env:

1. Low TiDB/TiFlash version
2. Restart TiFlash
3. Error happens when new TiFlash and old TiDB
4. Restart TiDB