Add board search command and gRPC interface function

[silvanocerza]

Please check if the PR fulfills these requirements

• The PR has no duplicates (please search among the Pull Requests
  before creating one)
• The PR follows our contributing guidelines
• Tests for the changes have been added (for bug fixes / features)
• Docs have been added / updated (for bug fixes / features)
• UPGRADING.md has been updated with a migration guide (for breaking changes)

• What kind of change does this PR introduce?

Adds a new CLI command and related gRPC interface function.

• What is the current behavior?

There is currently no way to get or search a list of boards from installed and installable platforms, manually installed platforms are included in the search.

• What is the new behavior?

A new board search command and its related gRPC interface can now be used to get a list of boards from installed and installable platforms.

Sample output:

$ arduino-cli board search                                                                                                                                                                                  
Board Name                                 FQBN Platform ID                                                                                                                                         
Adafruit Circuit Playground                     arduino:avr                                                                                                                                         
Adafruit Circuit Playground Express             arduino:samd                                                                                                                                        
Analog ADI                                      Arrow:samd                                                                                                                                          

Sample json output:

$ arduino-cli board search --format json
[
  {
    "name": "Arduino Mega ADK",
    "FQBN": "arduino:avr:megaADK",
    "platform": {
      "ID": "arduino:avr",
      "Installed": "1.8.3",
      "Latest": "1.8.3",
      "Name": "Arduino AVR Boards",
      "Maintainer": "Arduino",
      "Website": "http://www.arduino.cc/",
      "Email": "[email protected]"
    }
  },
...]

FQBN is omitted if a board's platform is not installed because that information is not present in the index.json file but only in the boards.txt file.

The result can be filtered passing one or more argument to the command, a fuzzy search is done to return all boards that closely match the arg string:

$ arduino-cli board search mkr1000                   
Board Name                          FQBN Platform ID  
Adafruit Circuit Playground Express      arduino:samd 
Arduino M0                               arduino:samd 
Arduino M0 Pro                           arduino:samd 
Arduino MKR FOX 1200                     arduino:samd 
Arduino MKR GSM 1400                     arduino:samd 
Arduino MKR NB 1500                      arduino:samd 
Arduino MKR Vidor 4000                   arduino:samd 
Arduino MKR WAN 1300                     arduino:samd 
Arduino MKR WAN 1310                     arduino:samd 
Arduino MKR1000                          arduino:samd 
Arduino MKRZERO                          arduino:samd 
Arduino Nano 33 IoT                      arduino:samd 
Arduino Tian                             arduino:samd 

• Does this PR introduce a breaking change, and is titled accordingly?

Nope.

• Other information:

None.

---

See how to contribute

[silvanocerza]

@kittaakos you probably want to test this.

[kittaakos]

The search result order is not deterministic. I get back the boards in a different order when searching with the same term: ard.

screencast.2021-03-04.14-35-36.mp4

[silvanocerza]

Yep, that's not good. I'd say I could order them alphabetically, first the installed ones then the rest.

[kittaakos]

Yep, that's not good. I'd say I could order them alphabetically

It would be already a great improvement.

first the installed ones then the rest.

I am not sure about this 👆 part.

[kittaakos]

I am hitting false-negative matches. For example, with the latest new IDE, I could find Arduino Duemilanove or Diecimila with ard due die. This does not work anymore.

This is how it works in the new IDE:

With the proposed change, no results:

The interesting part is that I have one hit for the term ard due, but I have zero for ard due die. Can you explain why it does not support ard due die?

The IDE2 will use this functionality as-is, so I think it make sense to discuss the requirements here.

[kittaakos]

I get unrelated matches. Search for NANO, you will see Uno for example:

./arduino-cli board search "NANO" --format json
[
  {
    "name": "Arduino Uno",
    "FQBN": "arduino:avr:uno",
    "platform": {
      "ID": "arduino:avr",
      "Installed": "1.8.3",
      "Latest": "1.8.3",
      "Name": "Arduino AVR Boards",
      "Maintainer": "Arduino",
      "Website": "http://www.arduino.cc/",
      "Email": "[email protected]"
    }
  },
  {
    "name": "Arduino Uno WiFi",
    "FQBN": "arduino:avr:unowifi",
    "platform": {
      "ID": "arduino:avr",
      "Installed": "1.8.3",
      "Latest": "1.8.3",
      "Name": "Arduino AVR Boards",
      "Maintainer": "Arduino",
      "Website": "http://www.arduino.cc/",
      "Email": "[email protected]"
    }
  },
  {
    "name": "Arduino Nano",
    "FQBN": "arduino:avr:nano",
    "platform": {
      "ID": "arduino:avr",
      "Installed": "1.8.3",
      "Latest": "1.8.3",
      "Name": "Arduino AVR Boards",
      "Maintainer": "Arduino",
      "Website": "http://www.arduino.cc/",
      "Email": "[email protected]"
    }
  },
  {
    "name": "Arduino Nano Every",
    "platform": {
      "ID": "arduino:megaavr",
      "Latest": "1.8.7",
      "Name": "Arduino megaAVR Boards",
      "Maintainer": "Arduino",
      "Website": "http://www.arduino.cc/",
      "Email": "[email protected]"
    }
  },
  {
    "name": "Arduino NANO 33 IoT",
    "FQBN": "arduino:samd:nano_33_iot",
    "platform": {
      "ID": "arduino:samd",
      "Installed": "1.8.11",
      "Latest": "1.8.11",
      "Name": "Arduino SAMD Boards (32-bits ARM Cortex-M0+)",
      "Maintainer": "Arduino",
      "Website": "http://www.arduino.cc/",
      "Email": "[email protected]"
    }
  },
  {
    "name": "Arduino Nano 33 BLE",
    "platform": {
      "ID": "arduino:mbed",
      "Latest": "1.3.2",
      "Name": "Arduino mbed-enabled Boards",
      "Maintainer": "Arduino",
      "Website": "http://www.arduino.cc/",
      "Email": "[email protected]"
    }
  }
]

Why is Arduino Uno in the resultset when the term is NANO? Is it due to the partial FQBN match arduiNo:Avr:uNO. Can we search for the name only? Or make it possible to search by name only via gRPC?

{
    "name": "Arduino Uno",
    "FQBN": "arduino:avr:uno",
    "platform": {
      "ID": "arduino:avr",
      "Installed": "1.8.3",
      "Latest": "1.8.3",
      "Name": "Arduino AVR Boards",
      "Maintainer": "Arduino",
      "Website": "http://www.arduino.cc/",
      "Email": "[email protected]"
    }
  }

[kittaakos]

This is rather a request and related to the NANO -> Uno question. When I search for mbed I can see the following boards.

I assume the match comes from the match on the ID of the platform:

[
  {
    "name": "Arduino Portenta H7",
    "platform": {
      "ID": "arduino:mbed",
      "Latest": "1.3.2",
      "Name": "Arduino mbed-enabled Boards",
      "Maintainer": "Arduino",
      "Website": "http://www.arduino.cc/",
      "Email": "[email protected]"
    }
  },
  {
    "name": "Arduino Nano 33 BLE",
    "platform": {
      "ID": "arduino:mbed",
      "Latest": "1.3.2",
      "Name": "Arduino mbed-enabled Boards",
      "Maintainer": "Arduino",
      "Website": "http://www.arduino.cc/",
      "Email": "[email protected]"
    }
  }
]

Can the gRPC API expose a flag to restrict the search to the name property only, please?

[kittaakos]

The IDE2 will use this functionality as-is, so I think it make sense to discuss the requirements here.

@ubidefeo, can you please chime in and check if this is what we need in the new IDE? Thank you!

[silvanocerza]

I am not sure about this 👆 part.

I won't do it then. 👍

Sadly the fuzzy search is kind of a double edge sword, thus the results above.

Why is Arduino Uno in the resultset when the term is NANO? Is it due to the partial FQBN match arduiNo:Avr:uNO.

Yes, it's probably that. The library I'm using for fuzzy searching returns the Levenshtein distance of the match, in this case I discard all results that have a distance greater than 20. Some times it's good, some times it's not, it all depends on the searched term and the boards' name/fqbn.

If I don't discard those searching for a small string like uno or similar would return basically all the boards, if I filter with a smaller distance I might miss some things. For example with a distance of 5 searching for uno wouldn't return Arduino Uno.

I know it's not perfect, I don't like it either, am thinking of reverting all the fuzzy search changes cause of this.

Can we search for the name only? Or make it possible to search by name only via gRPC?

I can do it but I'd like to keep the CLI and the gRPC interface aligned, so I'd do it for both.

[kittaakos]

Can we search for the name only
I can do it but I'd like to keep the CLI and the gRPC interface aligned

I think it would be useful to search the boards by only the name. I can understand the platform name, ID, and any board name match for the core search, but I would use the name only for the board search.I am not a tie-breaker, though.

[silvanocerza]

I've updated the PR, now the results are sorted alphabetically and the search arg acts only on the boards name.

[silvanocerza]

I've changed it again, it should be much more reliable now. Now it uses a similar logic used by the Java IDE libraries search, I enhanced it a bit so that accents and other similar diatrics are removed before trying to match the search query case insensitively; this way we can still match yun and Yún without returning unrelated boards.

Fuzzy search has been completely removed for this new command and I will do the same for other commands after this PR is merged.

@ubidefeo tested this using the CLI and is happy about the result, let me know if it's fine for the IDE too @kittaakos.

[kittaakos]

let me know if it's fine for the IDE too @kittaakos.

I am OK with the changes. The gRPC API remains the same no matter there is support for the fuzzy match or not.

[ubidefeo]

LGTM