Hive: close the fileIO client when closing the hive catalog

[hussein-awala]

This PR fixes the following warning:

WARN S3FileIO: Unclosed S3FileIO instance created by:
        org.apache.iceberg.aws.s3.S3FileIO.initialize(S3FileIO.java:359)
        org.apache.iceberg.CatalogUtil.loadFileIO(CatalogUtil.java:350)
        org.apache.iceberg.hive.HiveCatalog.initialize(HiveCatalog.java:111)
        ...

[nastra]

fileIO could be null at this point if initialization failed and close is called immediately after it, so it would be good to add a null check.

Generally speaking, there's no guarantee that adding close() here actually fixes the exception you mentioned in the PR. You would most likely need a similar concept to what was introduced by #7487 / #8315

[hussein-awala]

I just created a fileIOCloser to track the IO files and close them when the catalog is closed, could your recheck please?

[amogh-jahagirdar]

Looks like the flink test failure is unrelated, I reopened #10356 since it looks like that test is still flaky. I'm retriggering CI.

[amogh-jahagirdar]

Something I've wondered for a bit, is it possible to uplevel any of this logic to BaseMetastoreCatalog so that we avoid having every catalog which extends from that to have it's own Cache<TableOperations, FileIO>?

Or does that not generalize? Not a blocker, but just wanted to make sure we at least thought through it to avoid duplication.

[hussein-awala]

I tried to move it to uplevel but I found some subclasses without a FileIO instance (ViewAwareTableBuilder and HadoopCatalogTableBuilder), if you have any suggestions to generalize it, I'm open to it.

[nastra]

I have a TODO on my list to generalize this, so I'll take a look at this (hopefully this week)

[nastra]

@amogh-jahagirdar I've opened #10893 to addresse this

[amogh-jahagirdar]

Thanks @hussein-awala , and thank you @nastra for reviewing!