Azure MySQL SSL Connection Configuration (Metadata Store Server)

[andrijaperovic]

Have tried to modify the metadata-grpc service which is packaged in https://github.com/kubeflow/pipelines to pass the following config in text protobuf format to /bin/metadata_store_server:

connection_config {
      mysql {
        host: "..."
        port: int
        database: "..."
        user: "..."
        password: "..."
        ssl_options {
          verify_server_cert: true
          capath: "/etc/ssl/certs/my-cert.pem"
        }
      }
    }

Observing the following exception without much details:

2021-10-11 18:04:01.536992: F 
ml_metadata/metadata_store/metadata_store_server_main.cc:226] 
Non-OK-status: status status: Internal: mysql_real_connect failed: 
errno: 0, error: MetadataStore cannot be created with the given connection config.

To my knowledge, there is no way to enable additional verbosity on metadata_store_server.
As part of Kubeflow Pipelines 1.7.0-alpha.1, metadata-grpc-service is deployed using gcr.io/tfx-oss-public/ml_metadata_store_server:1.0.0 image.

We are using Azure Database for MySQL server, MySQL Version 5.7 with SSL Connections enforced.

[andrijaperovic]

The error message is not any more informative on image version 1.4.0:

WARNING: Logging before InitGoogleLogging() is written to STDERR
F1029 20:33:53.509305     1 metadata_store_server_main.cc:236] Check failed: absl::OkStatus() == status (OK vs. INTERNAL: mysql_real_connect failed: errno: , error:  [mysql-error-info='']) MetadataStore cannot be created with the given connection config.
*** Check failure stack trace: **

[BrianSong]

Duplicate issue as kubeflow/pipelines#6711

Please follow up there.

[andrijaperovic]

@BrianSong the duplicate issue was filed by me and there is no traction on the issue, I've already discussed with @berndverst it cannot be fixed in KFP. Since the KFP issue is directly due to a sub-component which is ml_metadata_store_server only, I think it can only be addressed in this repository directly.
Would you please kindly re-open this issue.

[andrijaperovic]

@BrianSong I've also reached out to @hughmiao on #20 since SSL seems to be supported in MySQL backend, however I'm not able to find documentation regarding a working config.

[BrianSong]

The SSLOptions are declare in [1].

[1] https://github.com/google/ml-metadata/blob/master/ml_metadata/proto/metadata_store.proto#L553

[andrijaperovic]

@BrianSong doesn't seem to be the case, if I modify the field to be SSLOptions instead of ssl_options I observe the following error:

[libprotobuf ERROR external/com_google_protobuf/src/google/protobuf/text_format.cc:317] 
Error parsing text-format ml_metadata.MetadataStoreServerConfig: 8:16: 
Message type "ml_metadata.MySQLDatabaseConfig" has no field named "SSLOptions".
2021-11-01 19:31:17.388144: F ml_metadata/metadata_store/metadata_store_server_main.cc:90] 
Non-OK-status: tensorflow::ReadTextProto(tensorflow::Env::Default(), filename, server_config) status: 
Data loss: Can't parse /config as text proto

EDIT:
Also, I've omitted capath in my current ssl_options as all ssl_options fields are optional in the config. So currently I am only passing verify_server_cert as true.

[BrianSong]

Yeah, your original config should be correct. One possible root cause here is the verify_server_cert field in ssl_options. See the comment in [1]. In the PR that add verify_server_cert[2], the author mention that "Without openssl 1.0.2 it won't work in some cases due to https://jira.mariadb.org/browse/MDEV-10594 (and its corresponding https://jira.mariadb.org/browse/CONC-250)." You can change the environment by editing the setup step for the image as instructed in [3].

[1] #20 (comment)
[2] #24
[3] #24 (comment)

[andrijaperovic]

Thanks @BrianSong .
Are you recommending that the base image of Dockerfile.manylinux2010 needs to be updated to have a newer version of openssl?
From the comment, the implication is that the base image is built using pypa/manylinux2010 + bazel version 0.24.1.

From the manylinux repository README it appears the manylinux2014 tags are the latest stable tags as manylinux2010 is EOL as of November 30th, 2020.
The only option I see is to use manylinux2014 which appears to have OpenSSL 1.0.2k-fips 26 Jan 2017.
I'm assuming this is the correct set of tags.

[BrianSong]

Change the manylinux2010 to manylinux2014 could be an option, but it may lead to other bugs as ml_metadata has not been tested in manylinux2014 yet.

Another option is to still use manylinux2010, but "edit the setup_environment to change its environment by installing packages, so the bazel_build after that line will have the new environment." as shown in [1].

[1] #24 (comment)

[BrianSong]

Also, I found someone raised a similar bug in [1]. In that issue description, the same error - "ml_metadata/metadata_store/metadata_store_server_main.cc:226] Non-OK-status: status status: Internal: mysql_real_connect failed: errno: 0, error: MetadataStore cannot be created with the given connection config." seems to be triggered by kubeflow side issue. Maybe you can mention it in the kubeflow issue you opened as well.

[1] kubeflow/pipelines#5872

[mansoshaik-deloitte]

@andrijaperovic Had you solved the SSL issue with Azure MySQL from metadata_store_server (metadata-grpc-server of Kubeflow)?

[andrijaperovic]

@mansoshaik-deloitte I have not, this issue was deprioritized, for now we have disabled required SSL connections to the backend MySQL since it goes over a private connection. Will take it up again at some point.

[nkosteski]

@andrijaperovic Answering for other folks that might be tracking this down. I think you were really close on this. I eventually got it working with the following proto configuration. Though I did have to mount the certificate which is mentioned here in the azure mysql documentation and point to that.

connection_config {
      mysql {
        host: "host_address"
        database: "metadb_database"
        user: "db_username"
        password: "db_password"
        ssl_options {
            ca: "/etc/ssl/certs/BaltimoreCyberTrustRoot.crt.pem"
        }
      }
    }