Speed up GetTables operation for Spark session catalog

[pan3793]

🔍 Description

Issue References 🔗

This pull request aims to speed up the GetTables operation for the Spark session catalog.
As reported in #4956, #5949, the GetTables operation is quite slow in some cases, and in #4444, kyuubi.operation.getTables.ignoreTableProperties was introduced to speed up the V2 catalog, but not covers session catalog.

Describe Your Solution 🔧

Extend the scope of kyuubi.operation.getTables.ignoreTableProperties to cover the GetTables operation for the Spark session catalog.

Currently, the basic step of GetTables in the Spark engine is

val catalog: String = getCatalog(spark, catalogName)
val databases: Seq[String] = sessionCatalog.listDatabases(schemaPattern)
val identifiers: Seq[TableIdentifier] = catalog.listTables(db, tablePattern, includeLocalTempViews = false)
val tableObjects: Seq[CatalogTable] = catalog.getTablesByName(identifiers)

then filter tableObjects with tableTypes: Set[String].

The cost of catalog.getTablesByName(identifiers) is quite high when the table number is large, e.g. dozen thousand.

For some cases, listing tables only for table name display, it is worth speeding up the operation while ignoring some properties(e.g. table comments) and query criteria(specifically in this case, when kyuubi.operation.getTables.ignoreTableProperties=true, criteria tableTypes will be ignored, and all tables and views will be treated as TABLE to return.)

Types of changes 🔖

• Bugfix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Test Plan 🧪

Pass GA

---

Checklist 📝

• This patch was not authored or co-authored using Generative Tooling

Be nice. Be informative.

[codecov-commenter]

Codecov Report

Attention: 7 lines in your changes are missing coverage. Please review.

Comparison is base (47a1091) 61.19% compared to head (058001c) 61.11%.
Report is 2 commits behind head on master.

Files	Patch %	Lines
...e/kyuubi/engine/spark/util/SparkCatalogUtils.scala	69.56%	5 Missing and 2 partials ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #6018      +/-   ##
============================================
- Coverage     61.19%   61.11%   -0.09%     
  Complexity       23       23              
============================================
  Files           623      623              
  Lines         37060    37103      +43     
  Branches       5024     5029       +5     
============================================
- Hits          22680    22674       -6     
- Misses        11948    11988      +40     
- Partials       2432     2441       +9     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[pan3793]

Merged to master/1.8