[HUDI-8427] Fix delete in file group reader and re-enable test in TestHoodieSparkMergeOnReadTableInsertUpdateDelete #12283
Conversation
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/HiveHoodieReaderContext.java
Show resolved
Hide resolved
github-actions
bot
added
size:S
| } | ||
| return value; | ||
| return (WritableComparable) HoodieRealtimeRecordReaderUtils.avroToArrayWritable(value, Schema.create(newType)); |
There was a problem hiding this comment.
No, it doesn't fix everything, but I think it might be worth it to fix that here as well
There was a problem hiding this comment.
Spent some time trying to fix 8261. I think it's too complex to get into 1.0.0
| @ParameterizedTest | ||
| @ValueSource(booleans = {true, false}) | ||
| public void testSimpleInsertUpdateAndDelete(boolean populateMetaFields) throws Exception { | ||
| Properties properties = populateMetaFields ? new Properties() : getPropertiesForKeyGen(); | ||
| properties.setProperty(HoodieTableConfig.PRECOMBINE_FIELD.key(), "timestamp"); |
There was a problem hiding this comment.
Does this mean that without preCombine field it does not work? I've marked this test to be revisited by HUDI-8551.
There was a problem hiding this comment.
the test data gen creates the record with an ordering field of type long. If precombine is not set, then we default to int in the filegroup reader. So then when we read the records we try to cast the long to an int which fails. timestamp field is long, so then we don't need to do any casting.
public RawTripTestPayload(String jsonData, String rowKey, String partitionPath, String schemaStr) throws IOException {
this(Option.of(jsonData), rowKey, partitionPath, schemaStr, false, 0L);
}
What is also interesting is that there are 3 more constructors for RawTripTestPayload. 2 of them take in any Comparable as the ordering value. The final one sets it as an int:
this.orderingVal = Integer.valueOf(jsonRecordMap.getOrDefault("number", 0L).toString());
Change Logs
since it is using eventtime based merging, we need to actually set a real field for the precombine.
Also need to convert the precombine to hive type when we read the hoodie records from log blocks
Impact
fix test, and also support precombine field types
Risk level (write none, low medium or high below)
low
Documentation Update
N/A
Contributor's checklist