name vs real_name inconsistency when executing lib commands

[kittaakos]

Bug Report

• The name of a library is the name of the folder in which the lib is contained: {directories.user}/Arduino/${name}
• The real_name of a library is the human-readable name of the lib. It comes from the library.properties.

This might be the desired behavior but it's strange. When I do a lib search or lib install the real_name is required, when I run a lib list, the name is required.

Current behavior

lib search works with real_name, not with name:

% ./arduino-cli lib search  AllThingsTalk_LoRaWAN_SDK               
No libraries matching your search.
Did you mean...
AllThingsTalk LoRaWAN SDK

% ./arduino-cli lib search "AllThingsTalk LoRaWAN SDK" 
Name: "AllThingsTalk LoRaWAN SDK"
  Author: AllThingsTalk
  Maintainer: Vanja <[email protected]>
  Sentence: LoRaWAN SDK for AllThingsTalk Maker
  Paragraph: Compatible with Microchip RN2483 and RN2903 devices.
  Website: https://github.com/allthingstalk/arduino-lorawan-sdk
  Category: Communication
  Architecture: *
  Types: Contributed
  Versions: [3.0.0, 3.1.0, 3.1.2, 3.1.3, 3.1.4, 3.1.5, 3.1.6]
  Provides includes: AllThingsTalk_LoRaWAN.h

lib install works with real_name but does not work with name:

% ./arduino-cli lib install AllThingsTalk_LoRaWAN_SDK 
Error resolving dependencies for AllThingsTalk_LoRaWAN_SDK: looking for library: library AllThingsTalk_LoRaWAN_SDK not found
% ./arduino-cli lib install "AllThingsTalk LoRaWAN SDK"
AllThingsTalk LoRaWAN SDK depends on AllThingsTalk LoRaWAN [email protected]
Downloading AllThingsTalk LoRaWAN [email protected]...
AllThingsTalk LoRaWAN [email protected] already downloaded
Installing AllThingsTalk LoRaWAN [email protected]...
Already installed AllThingsTalk LoRaWAN [email protected]

lib list works with name, but does not work with real_name:

% ./arduino-cli lib list AllThingsTalk_LoRaWAN_SDK
Name                      Installed     Available         Location Description                                                
AllThingsTalk_LoRaWAN_SDK 3.1.6         -                 user     LoRaWAN SDK for AllThingsTalk Maker                        

% ./arduino-cli lib list "AllThingsTalk LoRaWAN SDK"
No libraries installed.

I think this is a bug.

Expected behavior

All lib commands work either with both name and real_name or all require either the name or real_name.

Environment

• CLI version (output of arduino-cli version): arduino-cli Version: 0.13.0-rc1 Commit: ef57e49
• OS and platform: macOS Catalina (10.15.6 (19G2021))

Additional context

[per1234]

I ended up taking a close look at this situation while reviewing #1725. Here is a report on my findings:

Recommendations

Always use canonical library name

The library name is used as the sole unique identifier by the Arduino Library Manager system.

This name must be used whenever referring to the library in Arduino CLI output.

This name must be used whenever referring to the library in arguments to arduino-cli commands.

Record the canonical name when loading libraries

Data on each installed library is gathered during initialization.

Since there is some logic involved in determining the canonical name, it must be recorded at this time to avoid the need for code duplication by each consumer of the data:

• If the library has a library.properties metadata file (i.e., "1.5 format"), the library name is defined by the name field of library.properties.
• If the library does not have a library.properties metadata file (i.e., "1.0" or "legacy" format), the library name is defined by the folder name.

Redesign github.com/arduino/arduino-cli/arduino/libraries.Library

I believe the design of this struct is the source of some of the current problems, and will be the source of others in the future:

• Name: This is actually the base library folder name, so only the canonical library name for outdated "1.0 format" libraries
• RealName: This is the value of the name field of library.properties, so only the canonical library name for "1.5 format" libraries

The situation is made even more unintuitive by the fact that github.com/arduino/arduino-cli/arduino/libraries/librariesindex.Library.Name is the equivalent of github.com/arduino/arduino-cli/arduino/libraries.Library.RealName (and in this case is always the canonical name due to the index only containing "1.5 format" libraries.

Neither github.com/arduino/arduino-cli/arduino/libraries.Library.Name nor github.com/arduino/arduino-cli/arduino/libraries.Library.RealName serve as a single field that unconditionally provides canonical name.

The data they contain is useful in its own right. For example, they are both priority factors in library dependency resolution.

This means that an additional field must be added to store the canonical name. That solves the technical problem, but it will only make the human problem of the unintuitive struct design even worse. So the existing fields should be changed to have appropriate names.

The redesign of this important exported struct will be a breaking change, so it should be done during the current development phase where breaking changes are already commonplace.

Update the code for the new struct design

Examples:

• github.com/arduino/arduino-cli/arduino/libraries.Library.String
• High level canonicalization code in github.com/arduino/arduino-cli/cli/lib/installedResult.String

Examples of problems caused by the current system:

• Newer library is not listed as available #1041
• lib list and lib examples use incorrect library name #1577
• Incompatible architecture warning uses incorrect library name #1731
• https://forum.arduino.cc/t/serial-output-window-select-all-copy/1024251/4

These issues also affect Arduino IDE and Arduino Web Editor.

Demos

Set up

$ arduino-cli version
arduino-cli.exe  Version: nightly-20220512 Commit: b6d36c6 Date: 2022-05-12T01:37:29Z

$ arduino-cli lib install "Robot Motor"

compile

$ SKETCH_PATH="/tmp/LibNameBug"

$ mkdir "$SKETCH_PATH"

$ echo "#include <ArduinoRobotMotorBoard.h>
void setup() {}
void loop() {}
" > "$SKETCH_PATH/LibNameBug.ino"

$ arduino-cli compile -b arduino:megaavr:nona4809 -v "$SKETCH_PATH"

[...]

Alternatives for ArduinoRobotMotorBoard.h: [[email protected]]
ResolveLibrary(ArduinoRobotMotorBoard.h)
  -> candidates: [[email protected]]

[...]

WARNING: library Robot_Motor claims to run on avr architecture(s) and may be incompatible with your current board which runs on megaavr architecture(s).

[...]

Compiling library "Robot_Motor"

[...]

Using library Robot_Motor at version 1.0.3 in folder: E:\deleteme\arduino-cli\directories\user\libraries\Robot_Motor 

[...]

🐛 Library is referred to incorrectly as "Robot_Motor" throughout the compilation output.

lib examples

$ arduino-cli lib install "Robot Motor"
Downloading Robot [email protected]...
Robot [email protected] downloaded
Installing Robot [email protected]...
Installed Robot [email protected]

$ arduino-cli lib examples "Robot Motor"
No libraries found.

🐛 Command does not recognize correct library name.

lib list

See #932 (comment).

lib uninstall

$ arduino-cli lib install "Robot Motor"
Downloading Robot [email protected]...
Robot [email protected] already downloaded
Installing Robot [email protected]...
Installed Robot [email protected]

$ arduino-cli lib uninstall "Robot Motor"
Uninstalling [email protected]...

🐛 Library is referred to incorrectly as "Robot_Motor".

$ arduino-cli lib install "Robot_Motor"
Error installing Robot_Motor: Library 'Robot_Motor' not found

$ arduino-cli lib install "Robot Motor"
Downloading Robot [email protected]...
Robot [email protected] already downloaded
Installing Robot [email protected]...
Installed Robot [email protected]

$ arduino-cli lib uninstall "Robot_Motor"
Uninstalling [email protected]...

🐛 lib uninstall command accepts incorrect library names (in addition to the correct "Robot Motor"), but equivalent command lib install requires the use of the correct name.

lib upgrade

$ arduino-cli lib install "Robot Motor"@1.0.2
Downloading Robot [email protected]...
Robot [email protected] already downloaded
Installing Robot [email protected]...
Installed Robot [email protected]

$ arduino-cli lib upgrade
Downloading Robot [email protected]...
Robot [email protected] already downloaded
Installing Robot [email protected]...
Replacing [email protected] with Robot [email protected]...
Installed Robot [email protected]

🐛 Library is referred to incorrectly as "Robot_Motor", and inconsistently as well since it also referred to the library correctly as "Robot Motor" in the same sentence.

[cmaglie]

Fixed by #1878