Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make input tasks only require a JSON writer, not reader #1601

Merged
merged 21 commits into from
Dec 8, 2021
Merged

Conversation

lihaoyi
Copy link
Member

@lihaoyi lihaoyi commented Dec 7, 2021

Resolves #598. Input tasks results are re-evaluated every time and thus never read from the disk, and so a upickle.default.Reader is not required. Nevertheless, we still require that a upickle.default.Writer in order to generate the *.json files used for ./mill show and other debugging purposes.

Targeting #1600 to avoid merge conflicts

Let's see what CI says...

TBH not sure if this is worth doing, or whether we should just specify that Input tasks require a ReadWriter just to keep things consistent. We don't currently have any use case for de-serializing the input task JSON (e.g. even mill show takes the JSON from disk and spits it out verbatim without de-serializing) but that doesn't mean we won't find such use cases in future. But if such use cases do re-appear, we can always add that functionality back. Commands are write-only, so making Inputs write-only would also not be unprecedented

@lihaoyi lihaoyi changed the title Make input tasks only require a JSON writer, not reader [WIP] Make input tasks only require a JSON writer, not reader Dec 7, 2021
Comment on lines 22 to 30
def format = task match {
case t: Target[T] => Some(t.readWrite.asInstanceOf[upickle.default.ReadWriter[T]])
case _ => None
}
def writer = task match {
case t: mill.define.Command[T] => Some(t.writer.asInstanceOf[upickle.default.Writer[T]])
case t: Target[T] => Some(t.readWrite.asInstanceOf[upickle.default.ReadWriter[T]])
case _ => None
}
Copy link
Member Author

@lihaoyi lihaoyi Dec 7, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These pattern matches are kind of weird, and go against the un-sealed open nature of our Task class hierarchy. Thus I moved the logic into abstract methods on NamedTask.

But TBH I'm not sure if that's the right thing to do. We really should decide if Task is meant to be a closed sealed-trait/case-class hierarchy operated on with pattern matches or an open trait/class hierarchy operated on by inheritance and virtual method dispatch...

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Until now I always though of our tasks as a sealed hierarchy. We handle most of their specialities when evaluating them and I find it hard to come up with scenarios which could be realized by a new type of task, which can be solely realized through the tasks implementation. So we should make them sealed.

@lihaoyi lihaoyi changed the title [WIP] Make input tasks only require a JSON writer, not reader Make input tasks only require a JSON writer, not reader Dec 7, 2021
@lihaoyi
Copy link
Member Author

lihaoyi commented Dec 7, 2021

@lefou I reverted the pattern-match/virtual-methods change for now. It's orthogonal to the main goal of this PR, and I'm not yet confident enough to decide either way so easiest to just leave as is for now

Copy link
Member

@lefou lefou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I really like this change, as it removes an unnecessary restriction. And this issue already popped up in the chats.

I still have some questions to the implementation touched in this PR but not directly to this change. So feel free to address it or not. The Labelled case class has a writer method which is used to write, but it has also a format method. If I'm right, this one is only used to read stuff? I don't get why it is called format? Maybe you can take the opportunity to a) add proper return types (to make clear if we are also interested in the inferred writer-aspect or not) and b) probably rename it to read (or enlighten me why not).

Base automatically changed from stackable-overrides to main December 8, 2021 10:30
@lihaoyi
Copy link
Member Author

lihaoyi commented Dec 8, 2021

writer basically returns the Option[upickle.default.Writer[T]] for a particular Task[T], while format returns the Option[upickle.default.ReadWriter[T]].

The name format is a holdover from the play-json and other JSON libraries which called their ReadWriter types *Format.

We should clean it up, but I'd like to get this fix in first and we can clean up separately. As discussed, we should probably do something about the Task[T] hierarchy, which may make this particular issue moot.

Copy link
Member

@lefou lefou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@lihaoyi lihaoyi merged commit 82ec1de into main Dec 8, 2021
@lefou lefou added this to the after 0.10.0-M4 milestone Dec 8, 2021
@lefou lefou deleted the input-no-read branch December 8, 2021 10:50
pbuszka pushed a commit to pbuszka/mill that referenced this pull request Dec 26, 2021
…1601)

Resolves com-lihaoyi#598. Input tasks results are re-evaluated every time and thus never read from the disk, and so a `upickle.default.Reader` is not required. Nevertheless, we still require that a `upickle.default.Writer` in order to generate the `*.json` files used for `./mill show` and other debugging purposes.

Targeting com-lihaoyi#1600 to avoid merge conflicts

Let's see what CI says...

TBH not sure if this is worth doing, or whether we should just specify that `Input` tasks require a `ReadWriter` just to keep things consistent. We don't currently have any use case for de-serializing the input task JSON (e.g. even `mill show` takes the JSON from disk and spits it out verbatim without de-serializing) but that doesn't mean we won't find such use cases in future. But if such use cases do re-appear, we can always add that functionality back. `Command`s are write-only, so making `Input`s write-only would also not be unprecedented
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Input tasks require JSON readable and writable output
2 participants