Implementation of new pipeline interface

[mrchtr]

Draft implementation of the new pipeline interface:

• modified pipeline.py (introduce new interface itself)
• renaming dataset accordingly to the defined consumes and produces
• introduce schema which can be used inside of generic components

Note: Didn't fixed the test yet. Added a dummy test just to visualise how the pipeline could look like. First wanted to check if this goes into the right direction.

[PhilippeMoussalli]

Thanks Matthias! I think this is headed in the right direction :)
Left some comments

[PhilippeMoussalli]

General question

Do we need to return the class instance. So instead of this:

pipeline = Pipeline(
    pipeline_name="my_pipeline",
    pipeline_description="description of my pipeline",
    base_path="/foo/bar",
)

dataset = pipeline.read(
    name="load_images",
    schema={
        "image": type.binary  # or pa.binary()
    }
)

We would have:

pipeline = Pipeline(
    pipeline_name="my_pipeline",
    pipeline_description="description of my pipeline",
    base_path="/foo/bar",
)

pipeline.read(
    name="load_images",
    schema={
        "image": type.binary  # or pa.binary()
    }
)

A bit similar to the the way we used to add operations to the pipeline. I think both are valid but not sure which one is more intuitive

[mrchtr]

The idea behind returning the class instance was that to call apply on a dataset afterwards.
Returning the class instance gives the user the most flexibility. The user can decide which variant he wants to implement.

Either:

dataset = pipeline.read(...)
dataset = dataset.apply(...)

or

pipeline.read(...)
dataset.apply(...)

or even:

pipeline.read(...).apply(...).apply(...)

[PhilippeMoussalli]

ok that makes sense :)

[RobbeSneyders]

Also, the Dataset class can provide an interface for more interactive data exploration.

[PhilippeMoussalli]

I would add return an error if both consumes and produces are None

[mrchtr]

I would use the default values of the component specs if they are None.

[PhilippeMoussalli]

The schema here is mandatory I suppose

[PhilippeMoussalli]

Same comment as above, is the schema here just needed for renaming or does it also specify which fields to write?

[mrchtr]

The schema should specify which fields and belonging types will be written.

[PhilippeMoussalli]

Would docstring all the read/write/apply functions since they're user facing

[PhilippeMoussalli]

Shouldn't this schema be Dict[str, Type]? I would add some example on how this would look like in the docstring so the users know what to input and mainly because it's different from the write schema (this one would also benefit from having examples in the description), could you also move it to the top since it's mandatory?

[PhilippeMoussalli]

does the write requires a consumes?

[RobbeSneyders]

Closing in favor of #685