Simplify dataset format and interface

[RobbeSneyders]

We should revisit the design of the subsets and fields before moving forward with the implementation of any other composability-related issues.

Tasks

Preview Give feedback

1. Redesign dataset format and interface #567
   +0
2. Update core package #652
   +0
   
3. Update component package #655
   +0
   
4. Implement previous_index #656
   +0
5. Update components and manifests #642
   +0
6. Align ComponentHub readme generation to the new dataset interface #639
   +0
7. Align explorer on new dataset interface #640
   +0
8. Pass unconsumed columns when additionalFields is True #244
   Core +1
   
9. Map subset names between components #180
   Core +1
   
10. Dask merging dataframes experiment #649
    +0

[PhilippeMoussalli]

@RobbeSneyders should #244 and #180 be put on hold until we finish revisiting the subset and field design?

[RobbeSneyders]

Yes, I would do so.

[RobbeSneyders]

I moved everything related to the pipeline interface to #643. The documentation for both the dataset format (this issue) and the pipeline interface (#643) are bundled there.