[backend] mysql 8.0 support (upstream ml-metadata issue)

[thesuperzapper]

While we updated our clients for MySQL 8.0 in #8351, we still depend on google/ml-metadata, which does not fully support MySQL 8.0. In MySQL 8.0, the default-authentication-plugin was changed from mysql_native_password to caching_sha2_password, but ml-metadata does not support caching_sha2_password auth:

• fully support mysql 8.0 (caching_sha2_password authentication) google/ml-metadata#178

Furthermore, we are running a very old version of ml-metadata, specifically version 1.5.0 from 2021, but 1.13.1 exists, as raised here: #5439 (comment)

---

Impacted by this bug? Give it a 👍.

[zijianjoy]

We are hoping to be able to upgrade MLMD when its next minor version is available (1.14.0)

[davidspek]

@zijianjoy Just wanted to ping you since v1.14.0 has now been released.

[davidspek]

It looks like this was implemented in #9856, but that hasn't made it into a pipelines release yet.

[thesuperzapper]

@chensun just want to confirm if this important fix will be in the Kubeflow 1.8 version of Kubeflow Pipelines?

[chensun]

@chensun just want to confirm if this important fix will be in the Kubeflow 1.8 version of Kubeflow Pipelines?

#9856 is not included in any KFP release yet, so it won't be in Kubeflow 1.8 either.

[nithin8702]

We have upgraded mysql db from 5 to 8 and now its causing issue related to mlmd authentication plugin. i think kfp uses mysql_native_password type whereas mlmd requires caching_sha2_password. could anyone please how to overcome this? Is it possible to change mlmd to kfp like mysql_native_password ?

google/ml-metadata#178
#9549

@chensun @thesuperzapper

[papagala]

I'm confused by these two things at the same time, from the title of this issue

• we still depend on google/ml-metadata, which does not fully support MySQL 8.0. In MySQL 8.0, the default-authentication-plugin was changed from mysql_native_password to caching_sha2_password, but ml-metadata does not support caching_sha2_password auth from @thesuperzapper
• i think kfp uses mysql_native_password type whereas mlmd requires caching_sha2_password @nithin8702

I don't think they can both be true at the same time 🙃

Any help would be appreciated 😄

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[thesuperzapper]

This is absolutely critical, and should not be closed until it's resolved.

/lifecycle frozen

[nithin8702]

Any solution for this issue?

[thesuperzapper]

@nithin8702 you can continue to use mysql_native_password for your Kubeflow Pipelines user on your MySQL 8 database.

You just need to specify mysql_native_password when creating the user, as MySQL 8.0.4+ will use caching_sha2_password by default, which is not supported by Kubeflow Pipelines (specifically ml-metadata google/ml-metadata#178):

CREATE USER 'kubeflow'@'%' IDENTIFIED WITH mysql_native_password BY 'MY_PASSWORD';