[EPIC] Spark 3.4 Remaining Functionality

[NVnavkumar]

This epic encompasses remaining functionality to be implemented, and categorized to make it easier to organize work on these issues.

Categorized by related functionality

In these buckets, fixing one issue might help solve another issue in the same bucket.

URL Parsing (start with parse_url)

Both of these could probably be implemented with regular expressions or other string manipulation techniques.

• [FEA] Support parse_url  #6969
• [Audit][FEA][SPARK-39741][SQL] Support url encode/decode as built-in function #6023

ANSI

• [Audit][FEA] ANSI mode: CAST between AnsiIntervalType and IntegralType in Spark 3.4+ #5902
• [Audit][SPARK-39749][SQL] Always use plain string representation on casting Decimal to String #6015
• [AUDIT][SPARK-42045][SQL] ANSI SQL mode: Round/Bround should return an error on tiny/small/big integer overflow #7520

AQE

• [AUDIT][SPARK-40588] FileFormatWriter materializes AQE plan before accessing output ordering #7514
• [Audit][BUG] Ensure GPU handles user specified repartition in Spark 3.4 when AQE is enabled #6678
• [Audit][SPARK-41048][SQL] Improve output partitioning and ordering with AQE cache #7503

Datasource

Metadata

• [AUDIT] Add tests for FILE_BLOCK_START, FILE_BLOCK_LENGTH in metadata columns #7963
• [Audit][SPARK-41660][SQL] Only propagate metadata columns if they are used #7524
• [AUDIT][SPARK-41791] Add new file source metadata column types #7510

Other

• [FEA]- [Audit] [SPARK-38767] Support ignoreCorruptFiles and ignoreMissingFiles in Data Source options #5321
• [FEA] Support Empty2Null expression on Spark 3.4.0 #7043
• [FEA][SPARK-41096] Support reading parquet FIXED_LEN_BYTE_ARRAY type #7449
• [AUDIT][SPARK-41248][SQL] Add "spark.sql.json.enablePartialResults" to enable/disable JSON partial results #7482
• [AUDIT][SPARK-42521][SQL] Add NULLs for INSERTs with user-specified lists of fewer columns than the target table #8118

Plan output

• [SPARK-399776][SQL] JOIN verbose string should add Join type #6175
• [SPARK-39857][SQL] V2ExpressionBuilder uses the wrong LiteralValue data type for In predicate #6180

Error Handling

• [SPARK-39889][SQL] Use different error classes for numeric/interval divided by 0 #6199
• [Audit][BUG] Update GpuWindowSpecDefinition to use error classes instead of TypeCheckFailures in Spark 3.4 #6677

Categorized by type of fix

Most the issues here are only related by the type of work needed, and will not necessarily relate enough to make solving follow up work any easier.

CPU Fallback (not categorized)

• [SPARK-40308][SQL] Allow non-foldable delimiter arguments to str_to_map function #6543
• [FEA][SPARK-41462] Check whether supporting casting from Date and timestamp to TimestampNTZ #7478
• [AUDIT] [SPARK-41290][SQL] Support GENERATED ALWAYS AS expressions for columns in create/replace table statements #8113

Replicate Functionality

• [Audit][SPARK-41350][SQL][FOLLOWUP] Allow simple name access of join hidden columns after alias #7490
• [SPARK-39840][SQL][PYTHON] Factor PythonArrowInput out as a symmetry to PythonArrowOutput #6172
• [FEA][SPARK-39557][SQL] Support ARRAY, STRUCT, MAP types as DEFAULT values #6036

Validate Functionality

• [FEA][AUDIT][SPARK-42121][SQL] Add built-in table-valued functions posexplode, posexplode_outer, json_tuple and stack #8115