New installs overwrite existing libraries in the <sketchbook folder>/libraries

[KurtE]

Describe the bug
When I first installed I believe RC2 or 3 but for sure with Nightly build and started testing with the first versions of the
Teensy installs, I found I could no longer build some of the sketches I was working on.

I traced it down to the first run of the IDE, overwrote my existing libraries in my /libraries
and replaced the version I was working on.

In my Particular case it was the SD library. My current one was our current Github fork/branch of SD which is a thin wrapper to call off to the SDFat library.

Note: In my case, it was actually a symbolic link to where I keep all my sources
On my Windows machine I created this using: mklink /D SD d:\github\SD

Note: This also reproduced on my Ubuntu machine as well.

To Reproduce
Steps to reproduce the behavior:

Have some custom version of SD in your /libraries/SD
Install IDE 2
run the IDE

Also found that this will reproduce if you are having the issue, that updates in your tools.txt or tools.local.txt are not showing up as mentioned in:
arduino/arduino-cli#1614
And you delete the directory

C:\Users\<user name>\AppData\Roaming\arduino-ide\

And restart the IDE.

I noticed in the output window:

Downloading [email protected]
Installing [email protected]
Already installed [email protected]
...
Downloading [email protected]
Installing [email protected]
Replacing [email protected] with [email protected]
...

Notice it says it replaced SD and sure enough it did

Expected behavior
I would expect it to leave alone user libraries, especially ones marked with a higher version number

Screenshots

Desktop (please complete the following information):

• OS: Windows
• Version: 10
• OS: Ubuntu
• Version: 20.0.4

Additional context

[ubidefeo]

hey @KurtE

I'll bring this into our triaging, seems like a pretty annoying one.
Might be a regression introduced when we enabled first-install bundled libraries installation.

Thank you for reporting ✌️

[per1234]

The same happens with the first run "Arduino AVR Boards" installation (#497):

1. Use "Boards Manager" to install a previous version of the "Arduino AVR Boards" platform.
2. Select File > Quit from the Arduino IDE menus.
3. Rename or delete (:warning: cautiously) the following folder to simulate a first run of Arduino IDE 2.x:
   • Windows:

     C:\Users\<user name>\AppData\Roaming\arduino-ide
     

   • Linux:

     ~/.config/arduino-ide
     

   • macOS:

     ~/Library/Application Support/arduino-ide
     

4. Start the Arduino IDE.

The IDE will now upgrade the "Arduino AVR Boards" platform from 1.8.4 to 1.8.5.

Upgrading platform arduino:[email protected] with arduino:[email protected]

Some users may have been intentionally using a specific version of the platform with Arduino IDE 1.x in order to provide a controlled dependencies environment for their projects. They would likely not appreciate the IDE 2.x doing a silent upgrade when they make the migration.

[PaulStoffregen]

Is this issue still present in 2.0.0-rc7? (tagged "criticality: highest" in February)

[kittaakos]

I propose to check if this can be reproduced with the CLI only:

• wipe #directories/data and #directories/user/library,
• install a patched SD lib into #directories/user/library manually (verify you can use the library, compile!)
• install the AVR core with the CLI (no gRPC), and check if the SD was overridden.

Check if the AVR core has to be installed. The corresponding code is here.

[per1234]

install the AVR core with the CLI (no gRPC), and check if the SD was overridden.

The "Arduino AVR Boards" platform installation and the "SD" library upgrade are unrelated.

The "SD" library upgrade is caused by this: #663 (the gRPC equivalent of arduino-cli lib install Arduino_BuiltIn).

First run installation of "built-in" libraries upgrades previously installed versions

$ arduino-cli version
arduino-cli.exe  Version: 0.24.0 Commit: c1b10f56 Date: 2022-06-22T10:13:55Z

$ arduino-cli lib install [email protected]
[...]

$ arduino-cli lib list
Name Installed     Available     Location              Description
SD   1.2.3         1.2.4         LIBRARY_LOCATION_USER Enables reading and writing on SD cards.

$ arduino-cli lib install Arduino_BuiltIn

[...]

Installing [email protected]...
Replacing [email protected] with [email protected]...
Installed [email protected]

[...]

$ arduino-cli lib list
Name            Installed Available Location              Description
Arduino_BuiltIn 1.0.0     -         LIBRARY_LOCATION_USER -
Ethernet        2.0.0     -         LIBRARY_LOCATION_USER -
Firmata         2.5.8     -         LIBRARY_LOCATION_USER -
Keyboard        1.0.4     -         LIBRARY_LOCATION_USER -
LiquidCrystal   1.0.7     -         LIBRARY_LOCATION_USER -
Mouse           1.0.1     -         LIBRARY_LOCATION_USER -
SD              1.2.4     -         LIBRARY_LOCATION_USER -
Servo           1.1.8     -         LIBRARY_LOCATION_USER -
Stepper         1.1.3     -         LIBRARY_LOCATION_USER -
TFT             1.0.6     -         LIBRARY_LOCATION_USER -

So this is really a deficiency in the Arduino CLI capabilities (no way to specify that lib install should not upgrade). However, this will no longer be a concern after #1055

First run installation of "Arduino AVR Boards" platform upgrades previously installed versions

$ arduino-cli version
arduino-cli.exe  Version: 0.24.0 Commit: c1b10f56 Date: 2022-06-22T10:13:55Z

$ arduino-cli core update-index

[...]

$ arduino-cli core install arduino:[email protected]

[...]

$ arduino-cli core list
ID          Installed Latest Name
arduino:avr 1.8.4     1.8.5  Arduino AVR Boards

$ arduino-cli core install arduino:avr


[...]

Upgrading platform arduino:[email protected] with arduino:[email protected]...
Configuring platform....
Platform arduino:[email protected] installed

$ arduino-cli core list
ID          Installed Latest Name
arduino:avr 1.8.5     1.8.5  Arduino AVR Boards

So this is really a deficiency in the Arduino CLI capabilities (no way to specify that core install should not upgrade).

Conclusion

My proposal is to add an upgrade field to the PlatformInstall request message of the Arduino CLI gRPC interface:

https://arduino.github.io/arduino-cli/dev/rpc/commands/#cc.arduino.cli.commands.v1.PlatformInstallRequest

Once that is in place, it will be trivial to change Arduino IDE's first run installation of the "Arduino AVR Boards" platform to set that field to false, so that the platform will only be installed if it is not already present on the user's system. I would be happy to submit an issue to track that.

Even if not absolutely needed by Arduino IDE 2.x, it would be best to do the same in Arduino CLI for the LibraryInstall request message, and that could also be utilized in a quick fix for the library upgrade behavior of Arduino IDE's first run built-in library installation if #1055 will be delayed.

What do you think @kittaakos?