Core, Spark: Refactor RewriteFileGroup planner to core

[pvary]

As requested in #11497 extracted the Spark and Core related changes.
The change refactors the Spark RewriteFileGroup related code to core (RewriteFileGroupPlanner), so Flink could later reuse it.

[pvary]

Removed the serializable as it was not required by the tests.
@RussellSpitzer, @szehon-ho: Could you please check if this is ok?

[RussellSpitzer]

I would just hesitate to remove something that was there before. I think originally we wanted it because we were considering shipping this class around but I don't see any evidence that it is currently serialized.

[RussellSpitzer]

Now that these are Public we need java docs describing their purpose

[pvary]

I will add the javadoc.
Since we make this public, I would remove the Serializable. We can always add back if needed, but I restrict ourselves on a public API only when it is strictly necessary.

[pvary]

Added the javadoc.

I think shipping around a stream is a non-trivial task, so I would keep it not Serializable for now.

[pvary]

This is somewhat unrelated change - Flink will need to extract the split size since in the Flink compaction implementation the planner which needs its own rewriter, and the executor (where the rewriter is not used) runs on a different instance.
We could put into the FileGroupInfo, but as a first run I opted for the minimal change, especially on public APIs.

[RussellSpitzer]

For easier review it would be great if you could highlight the changes made.

I see the note on RewritePlanResult but i'm not sure where this class came from. It is unclear what has been extracted and renamed and what is new code.

[RussellSpitzer]

I'm a little confused about the inheritance structure here. If we want to make this common, maybe we should just have this belong to the FileRewriter as a base class?

If I can think through this correctly in this draft we have

RewriteFileGroupPlanner Responsible for

1. Scanning Table
2. Grouping Tasks by Partition
3. Further Grouping and Filtering in FileRewriter

FileRewriter Responsible for

1. Filtering Tasks based on Rewriter Config
2. Grouping tasks within partitions

Feels like all of that could just be in the FileRewriter without having a separate GroupPlanner? Do we have an argument against that? I think we could even just have some new default methods on the FileRewriter interface for planning?

[pvary]

Let me check that. I was hesitant to change existing classes, but if we have a consensus, I'm happy to do that.

[pvary]

The FileRewriter is a generic class with <T extends ContentScanTask<F>, F extends ContentFile<F>>.
The RewriteFileGroup which is returned by the planning is specific for FileScanTask.

I'm not sure that we use the FileRewriter with any other parametrization ATM, but to put everything into a single class we need to do one of the things below:

1. Changing FileRewriter<T extends FileScanTask> instead of the current FileRewriter<T extends ContentScanTask<F>, F extends ContentFile<F>>, or
2. Changing the public List<FileScanTask> RewriteFileGroup.fileScans() to return public <T extends ContentScanTask<F>, F extends ContentFile<F>> List<T> fileScans

Both of the would be a breaking change in the public API so we should be careful about them.

We still can "duplicate" and create a new FileRewriter, and deprecate the old one - but I would only do this, if we think that the original API needs a serious refactor anyways.

WDYT?

[RussellSpitzer]

I've been thinking a lot about the refactor. Trying to figure out what we should do.

we could possibly do

Rewriter[T]
has a Planner[T] // Responsible for generating tasks (move actual filtering into rewriter)
Maybe we should also have

Planner[T]
    DeletePlanner[DeleteScanTask]
    FilePlanner[FileScanTask]

So then you would have

SizeBasedDataRewriter <FileScanTask, DataFile>
   private planner = new FilePlanner<FileScanTask>

Would that end up being more complicated? I feel like we are getting some interdependencies here I'm not comfortable with extending. @aokolnychyi Probably has some smart things to say here as well

[RussellSpitzer]

Minor fix; we can drop this now and use EmptyStructLike.get()

[pvary]

Do we want to make EmptyStructLike, and EmptyStructLike.get public?
Currently it is package private.

[pvary]

Made EmptyStructLike public for now, but if you think this change doesn't worth it to make it public, I can revert this part.

[pvary]

Got this exception:

TestRewriteDataFilesAction > testBinPackRewriterWithSpecificOutputSpec() > formatVersion = 3 FAILED
    java.lang.UnsupportedOperationException: Can't retrieve values from an empty struct
        at org.apache.iceberg.EmptyStructLike.get(EmptyStructLike.java:40)
        at org.apache.iceberg.types.JavaHashes$StructLikeHash.hash(JavaHashes.java:96)
        at org.apache.iceberg.types.JavaHashes$StructLikeHash.hash(JavaHashes.java:75)
        at org.apache.iceberg.util.StructLikeWrapper.hashCode(StructLikeWrapper.java:96)
        at java.base/java.util.HashMap.hash(HashMap.java:338)

So reverted the EmptyStructLike changes

[RussellSpitzer]

Blargh

[pvary]

For easier review it would be great if you could highlight the changes made.

I see the note on RewritePlanResult but i'm not sure where this class came from. It is unclear what has been extracted and renamed and what is new code.

Let me amend this now, and thanks for taking a look anyway!

So there were no significant new code in the refactor.
Created a single new class (RewriteFileGroupPlanner) for the functionality, and created a new class (RewritePlan) for storing result. The responsibility of the new class is to group the rewrite tasks together and return the calculated plan.

Originally the RewriteDataFilesSparkAction.toGroupStream did the grouping. It used several other methods (planFileGroups, groupByPartition, fileGroupsByPartition, newRewriteGroup) and the RewriteExecutionContext inner class to do the grouping.

The methods and the class got moved to the RewriteFileGroupPlanner. The result got its own class to be able to return not only the stream, but the total counts as well which are needed by the current Spark implementation.

[pvary]

@RussellSpitzer: This is ready to another round if you have time.
Thanks,
Peter

[szehon-ho]

Looks reasonable to me. Some minor nit

[RussellSpitzer]

@pvary and I were talking this over a bit, I think we really want to get a stronger division between "Planning" and "Excecution" since the two are very intertwined right now.

Ideally we end up in a situation where core has base classes that are only responsible for planning out exactly which files should be rewritten and in what logical groups they should be rewritten in, while another class is responsible for the physical implementation on how that actually occurs. Currently I think our structure is: Action contains Rewriter which extends a Planning Class with an Implementation. Because of this the Action and rewriter talk back and forth causing planning to occur in both the action and with parts of code in the rewriter.

[pvary]

This is just a WIP for the change, which uses git file renames. I hope it is easier to review this way.

After the initial reviews, I need to do these steps

• Instead of the renames, deprecate the old API, and create a new one with these classes
• Testing needs to be reviewed (kept the current ones, so I know that I don’t break known features)
• CI and gradle changes can be reverted (these are needed now because of the renames)

[RussellSpitzer]

Write now this method is just used to set config from the plan in the rewrite executor, i'm wondering if we should just be setting those parameters directly on the rewriter rather than inheriting them from the plan?

[pvary]

If we are fairly confident, that only writeMaxFileSize and outputSpecId comes from the plan, then we can add setWriteMaxFileSize and setOutputSpecId to the FileRewriteExecutor API instead of the initPlan(FileRewritePlan)