[VL] Result mismatch issues Tracker

[zhouyuan]

Backend

VL (Velox)

Bug description

There are several data mismatch issues either related with operator or functions. Some of the fixes are landed in Gluten, and some are in Velox repo.
We will use this issue to track the status as these are critical for production envs.

• complex datatype return wrong value, disabled in Gluten now
• parquet scan + filter pushdown wrongly return "", should return null.
• distinct hash agg + spill returned duplicated keys.
• max_by function return wrong result
• sortBeforeRepartition
• cast(sum(decmial(20,4)), float)
• get_json_object({"dScore":0.0215434648799772}, "$.dScore") (Fixed in Velox)
• cast(string as bigint)
• cast(double as decimal)
• array_size(null) (issue: [VL] array_size(null) results inconsistent with vanilla spark #5248)
• round(avg(cast(col as double)), 4) [VL] Results are mismatch with Vanilla Spark when round(avg(cast(col as double)), 4) in release-1.1 #5366
• isNull and isNotNull in filter condition [GLUTEN-5682][VL] Fix incorrect result when isNull & isNotNull coexist in filter #5670
• from_unixtime with overflowed (Fixed by Add allowOverflow flag to Timestamp::toTimezone facebookincubator/velox#9836)
• date_format ([VL] date_format returns wrong results #5524)
• cast integer as binary ([VL] Casting integer to binary type produces inconsistent result with Spark #5073)
• regexp_replace('a{bc', '\{', '\[') ([VL] Results are mismatch with vanilla Spark when use regexp_replace('a{bc', '\\{', '\\[') #6224)
• LEGACY timeParserPolicy ([VL] Results are mismatch with vanilla Spark when using unix_timestamp(yyyy-MM-dd HH:mm:ss,SSS) #6227)
• FlushableAgg ([VL] Result mismatch found in FlushableAgg #6630)
• if expression ([VL] Results mismatch when scan low version orc file #6673)
• weekOfYear (Result mismatch of weekOfYear #6784)
• round ([VL] result mismatch found in round #6827)
• cast string as date ([VL] Result missmatch in cast string as date #6828)
• nested decimal arithmetic expressions ([VL] Incorrect result type of nested decimal arithmetic expressions #7082)
• date format week year ([VL] Result mismatch in date format week year #7069)
• Large timestamp outside of range ([VL] Large timestamp outside of range #7109)
• Aggregate window gets the wrong result ([VL] Aggregate window gets the wrong result in 1.2.0 #7194)
• Diff of in_or_and ([VL] Diff of in_or_and #7362)
• hash agg output wrong result ([VL] Diff on parquet agg #7494)
• cast double as string ([VL][1.2] Result of cast(double as string) is inconsistent with vanilla spark #7730)
• Cast string to integral type does not ignore ISO control characters ([VL][1.2][Result mismatch] Cast string to integral type does not ignore ISO control characters #7749)
• get_json_object ([VL][1.2] Result mismatch of get_json_object when json string has newline #7777)

[FelixYBW]

#4678 issue in hashagg

[FelixYBW]

#4587

Currently we disabled all complex data read

[zhouyuan]

#4818

[zhouyuan]

#4872

[kecookier]

#4891

[kecookier]

#4928

[kecookier]

#4930

[kecookier]

#4947

[FelixYBW]

3 issues we met:

1. parquet scan + filter pushdown wrongly return "", should return null. Fixed by Fix read parquet for different encodings across row groups facebookincubator/velox#9129
2. distinct hash agg + spill returned duplicated keys. distinct hash aggregate returned duplicated value if spill happens facebookincubator/velox#9219
3. max_by function return wrong result

[ulysses-you]

2. distinct hash agg + spill returned duplicated keys.

@FelixYBW Has this issue not been fixed by #4443 ?

[FelixYBW]

@FelixYBW Has this issue not been fixed by #4443 ?

No, it's tested from main branch. A new issue

[FelixYBW]

No, it's tested from main branch. A new issue

facebookincubator/velox#9219

[FelixYBW]

3. max_by function return wrong result

@yma11 Did you submit a fix to the issue?

[yma11]

3. max_by function return wrong result

@yma11 Did you submit a fix to the issue?

Not yet. Only have pushed to golden branch and will submit one in Velox upstream.

[NEUpanning]

#5253

[FelixYBW]

#5253

Looks the issue of get_json_object. @PHILO-HE maybe we need a fully tests of json functions, like the re2.

[PHILO-HE]

#5253

Looks the issue of get_json_object. @PHILO-HE maybe we need a fully tests of json functions, like the re2.

@FelixYBW, I will do that. Thanks!

[FelixYBW]

#5248

[kecookier]

#5366

[FelixYBW]

#5366

UPdated desc. thank you. do you know which function (cast, avg, round ) caused the issue?

[FelixYBW]

#5372

[yma11]

3. max_by function return wrong result

@yma11 Did you submit a fix to the issue?

Not yet. Only have pushed to golden branch and will submit one in Velox upstream.

@FelixYBW This fix should be done at cpp side. The formal fix is in PR. Can you help review it?

[FelixYBW]

@FelixYBW This fix should be done at cpp side. The formal fix is in PR. Can you help review it?

Is it a Gluten issue? I'd think veox has some bug here.

[yma11]

@FelixYBW This fix should be done at cpp side. The formal fix is in PR. Can you help review it?

Is it a Gluten issue? I'd think veox has some bug here.

Yes. I think so. It's caused by the additional projects we added before/after shuffle. The logic of partial/final handle in Velox upstream has no problem. The ideal way is to add struct support for shuffle in Gluten so that we can remove the hack.

[FelixYBW]

@PHILO-HE Any update of the issues here?

[zjuwangg]

#5682

[NEUpanning]

#5701

[PHILO-HE]

@PHILO-HE Any update of the issues here?

@FelixYBW, Some were actually fixed. Just updated the list. Will fix or seek help to fix other issues.

[kecookier]

#6224

[NEUpanning]

#6227

[kecookier]

#6630

[NEUpanning]

#6673

[zml1206]

#6784

[jiangjiangtian]

#6827

[zml1206]

#6828

[NEUpanning]

#6845

[FelixYBW]

#6845

Closed. Not a bug.

[jiangjiangtian]

#7082

[kecookier]

#7069

[ccat3z]

#7109

[kecookier]

#7194

[FelixYBW]

#7494

[FelixYBW]

#7730

[wForget]

#7749

[wForget]

#7777