[FEA] Reimplement $ transpilation using cuDF new line terminator support

[NVnavkumar]

Is your feature request related to a problem? Please describe.

cuDF added support for multiple new-line characters in rapidsai/cudf#15961, which allows support for the different Java unicode line terminator characters. This requires passing a flag to the cuDF regex APIs to enable this mode, and updating the transpiler to a more simplified implementation of $ (which only needs to add support for the \r\n combination in addition to the individual characters already supported by cuDF:

• \n line-feed (already supported)
• \r carriage-return
• \u0085 next line (NEL)
• \u2028 line separator
• \u2029 paragraph separator

Additional context

This might fix failing tests here:

• [BUG] test_regexp_choice failed #10641
• [BUG] Integration test test_re_replace_all fails with a corner case #9731

Also, can look into a possible solution for:

• [FEA] Support single '$' or '^' on right side of regexp choice  #10764

[SurajAralihalli]

We should start by adding EXT_LINE to the RegexFlag.java file to enable support for multiple newline characters to JAVA APIs, as introduced in rapidsai/cudf#15961. See RegexFlag.java, line 27, small adjustments will be needed in the cuDF repo.

Update: PR #17139

After this, we can begin migrating the Spark Regex APIs. This involves updating the transpiler function in the RegexParser to remove the workaround used for previous cuDF’s multiple line delimiter limitation, see RegexParser.scala#L850. Additionally, we must modify all Spark GPU Regex API expressions, such as GpuRegExpExtract, to incorporate cuDF’s fix. See stringFunctions.scala#L1419:GpuRegExpExtract.