ddl: Fix potential data lost of alter_partition_by (release-7.5)

[JaySon-Huang]

cherry-pick of #8337

What problem does this PR solve?

Issue Number: close #8206

Problem Summary:

Introduced by #7822

When executing alter table xxx partition by ... to turn a non-partition table into partition table, there could be a chance that tiflash see a non-partition table turn into a partition table (using the same table_id). But it would be skipped by the previous implementation

tiflash/dbms/src/TiDB/Schema/SchemaBuilder.cpp

Lines 407 to 423 in b092db0

	template <typename Getter, typename NameMapper>
	void SchemaBuilder<Getter, NameMapper>::applyPartitionDiff(
	const TiDB::DBInfoPtr & db_info,
	const TableInfoPtr & table_info,
	const ManageableStoragePtr & storage)
	{
	const auto & orig_table_info = storage->getTableInfo();
	if (!orig_table_info.isLogicalPartitionTable())
	{
	LOG_ERROR(
	log,
	"old table in TiFlash not partition table {} with database_id={}, table_id={}",
	name_mapper.debugCanonicalName(*db_info, orig_table_info),
	db_info->id,
	orig_table_info.id);
	return;
	}

And then tiflash mistaken drop the old table along with all its partitions, but actually those partition are now attached to a new logical table. This leads to data lost after `alter table xxx partition ...

What is changed and how it works?

• Use tidb_isolation_read_engines instead of hint in test cases
• Allow turning a non-partition table into partition table
• When applying SchemaDiff for alter partition, we first create the new table and override the partition id mapping before dropping the old table

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No code

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

None

[JaySon-Huang]

/run-all-tests

[Lloyd-Pottiger]

Suggested change

	// If table is partition table, we will create the logical table here.
	// If table is partition table, we will create the physical table here.

typo?

[JaySon-Huang]

The applyCreatePhysicalTable only create the IStorage for logical table. The function name is kind of misleading.

tiflash/dbms/src/TiDB/Schema/SchemaBuilder.cpp

Lines 940 to 1018 in 6999a2f

	template <typename Getter, typename NameMapper>
	void SchemaBuilder<Getter, NameMapper>::applyCreatePhysicalTable(
	const TiDB::DBInfoPtr & db_info,
	const TableInfoPtr & table_info)
	{
	GET_METRIC(tiflash_schema_internal_ddl_count, type_create_table).Increment();
	LOG_INFO(
	log,
	"Creating table {} with database_id={}, table_id={}",
	name_mapper.debugCanonicalName(*db_info, *table_info),
	db_info->id,
	table_info->id);
	/// Check if this is a RECOVER table.
	{
	auto & tmt_context = context.getTMTContext();
	if (auto * storage = tmt_context.getStorages().get(keyspace_id, table_info->id).get(); storage)
	{
	if (!storage->isTombstone())
	{
	LOG_DEBUG(
	log,
	"Trying to create table {}, but it already exists and is not marked as tombstone, database_id={} "
	"table_id={}",
	name_mapper.debugCanonicalName(*db_info, *table_info),
	db_info->id,
	table_info->id);
	return;
	}
	LOG_DEBUG(
	log,
	"Recovering table {} with database_id={}, table_id={}",
	name_mapper.debugCanonicalName(*db_info, *table_info),
	db_info->id,
	table_info->id);
	AlterCommands commands;
	{
	AlterCommand command;
	command.type = AlterCommand::RECOVER;
	commands.emplace_back(std::move(command));
	}
	auto alter_lock = storage->lockForAlter(getThreadNameAndID());
	storage->updateTombstone(
	alter_lock,
	commands,
	name_mapper.mapDatabaseName(*db_info),
	*table_info,
	name_mapper,
	context);
	LOG_INFO(
	log,
	"Created table {}, database_id={} table_id={}",
	name_mapper.debugCanonicalName(*db_info, *table_info),
	db_info->id,
	table_info->id);
	return;
	}
	}
	/// Normal CREATE table.
	if (table_info->engine_type == StorageEngine::UNSPECIFIED)
	{
	auto & tmt_context = context.getTMTContext();
	table_info->engine_type = tmt_context.getEngineType();
	}
	String stmt = createTableStmt(*db_info, *table_info, name_mapper, log);
	LOG_INFO(
	log,
	"Creating table {} (database_id={} table_id={}) with statement: {}",
	name_mapper.debugCanonicalName(*db_info, *table_info),
	db_info->id,
	table_info->id,
	stmt);
	ParserCreateQuery parser;
	ASTPtr ast = parseQuery(parser, stmt.data(), stmt.data() + stmt.size(), "from syncSchema " + table_info->name, 0);

[Lloyd-Pottiger]

get

[JaySon-Huang]

/run-all-tests

[JaySon-Huang]

/run-all-tests

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: hongyunyan, Lloyd-Pottiger

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [Lloyd-Pottiger,hongyunyan]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2023-11-10 02:30:51.070559569 +0000 UTC m=+3784248.657669699: ☑️ agreed by Lloyd-Pottiger.
• 2023-11-10 06:04:22.415377633 +0000 UTC m=+3797060.002487777: ☑️ agreed by hongyunyan.