Binaries export must now be explicitly specified

[silvanocerza]

Please check if the PR fulfills these requirements

• The PR has no duplicates (please search among the Pull Requests
  before creating one)
• The PR follows
  our contributing guidelines
• Tests for the changes have been added (for bug fixes / features)
• Docs have been added / updated (for bug fixes / features)

• What kind of change does this PR introduce?

A change to an existing feature.

• What is the current behavior?

Currently, if the --build-path is not specified, compilation is done using a temporary folder generated deterministically given the absolute path of the Sketch selected; the generated binaries are then copied from that folder to <sketch_folder>/build/<fqbn>/.
Using the --dry-run flag the user can stop the CLI from copying the binaries from the build folder to the sketch one.

To upload a sketch compiled using the --dry-run flag the user must specificy the build folder in the temp directory with the --input-dir flag or run again compilation omitting --dry-run, otherwise it will fail.

• What is the new behavior?

The --dry-run flag is removed.
compile never copies generated binaries to the sketch folder.
The --export-binaries (-e) flag is introduced to copy the binaries from the build folder to the sketch one.
Specifying the --output-dir doesn't require using setting also the --export-binaries flag.
A related settings and env var has been added to avoid the need to always specify the --export-binaries flag: sketch.always_export_binaries and ARDUINO_SKETCH_ALWAYS_EXPORT_BINARIES.

If --input-dir or --input-file is not set when calling upload the command will search for the deterministically created build directory in the temp folder and use the binaries found there.

The gRPC interface has been updated accordingly, dryRun is removed.
I also removed exportFile since it was unused.

• Does this PR introduce a breaking change?

Yes, this introduces some breaking changes.

• Other information:

This will also help avoid commiting binaries to a Sketch repository by mistake since no binaries will be in that folder.

---

See how to contribute

[matthijskooijman]

Sounds like a good change to me. Dry-run was a bit weird, since it did actually do stuff (copy files to build dir, run compilers, etc.).

[facchinm]

A note here: to handle correct deprecation policies and binary compatibility the ID of exported grpc functions should never change, only increment and/ore remove.
So if the plan is to remove string exportFile = 13 simply delete that line

[silvanocerza]

Good to know, I had some doubt about it in fact. I'll fix it.

[facchinm]

Still not ok :) export_dir should be 18, and clean 19, while export_binaries becomes 20

[silvanocerza]

🤦 I'll get to it.

[ubidefeo]

Works as expected, but I am afraid because of the default behaviour some users' workflows will break.
If a user relies on the currently automatically created build folder to do whatever, then as this gets merged and ends up in nightly we'll have to face some backlash

I do like the behaviour, though.
One silly question: what are the chances that this fails?

the command will search for the deterministically created build directory in the temp folder and use the binaries found there.

[facchinm]

@ubidefeo

One silly question: what are the chances that this fails?

Same chance of md5 clash on two strings (I agree that md5 is not ideal but every other hash would make the folder name too long with the usual Windows problems we are aware of)

arduino-cli/arduino/builder/builder.go

Line 35 in d83170e

md5SumBytes := md5.Sum([]byte(path))

[silvanocerza]

Chance of an md5 clash are pretty slim I think, how many sketches you'd have to compile for that chance?

[ubidefeo]

@facchinm @silvanocerza
my concern wasn't name clashing, but how repeated compile/upload (and the new --clean flag) might play together.
Isn't this temporary folder name going to change?
I don't exactly know what it's based on, and how the CLI knows which target folder to keep using if we're taking advantage of cached objects

[silvanocerza]

The folder name in the temp dir has this structure arduino-sketch-<md5hash>, the md5 is calculated from the absolute path of the compiled sketch so if you move the sketch or rename it you should compile it again otherwise upload fails.
The --clean flag just deletes all existing files in the build directory so I don't expect it to break anything.