[CI] External libraries compilation test (#8003)

* Workflow final + docs added

Delete LIBRARY_TEST.md

Update readme + documentation

Generated External Libraries Test Results

Updated docs

Generated External Libraries Test Results

revert last commit

DOnt install arduino-core manually

Update external_libraries_test.rst

Generated External Libraries Test Results

Align FQBNs to platform name

use manula platform installation

Generated External Libraries Test Results

remove verbose

remove - before sketch path

Add libs and examples for specific chip

Generated External Libraries Test Results

Generated External Libraries Test Results

Update README.md

Generated External Libraries Test Results

Update README.md

Update README.md

Generated External Libraries Test Results

change name of GH action

replace - for /

Generated External Libraries Test Results

Generated External Libraries Test Results

Update lib.yml

Update lib.yml

Generated External Libraries Test Results

remove types from pull_rewuest

add pull to lib.yml

Generated External Libraries Test Results

add back deltas_report

Update LIBRARY_TEST.md

add verbose logs

remove deltas report

Generated External Libraries Test Results

Change from schedule to push for testing

add P-R-O-C-H-Y repo to cron job

rename workflow

Update LIBRARY_TEST.md

Update LIBRARY_TEST.md

Edit URL

Generated External Libraries Test Results

remove unused ymls + run job on schedule

generated

Add action link

push only result file

generated

fix names

fix workflow

run workflow on push

Add report to file action

remove separator

fix example paths

fix library name

yaml list of examples edit

update universal sketches

examples path fix

update example paths

Add ESP32Servo

warnings -> all

change warnings to "default"

fix library name

fix list

add matrix to compile sketches

test compile for only 1 chip

remove GPIO example

use edited action

Add Blynk library

use v1

print arduino cli version

fix listing

update libs

list fix

remove if

fix typo

fix path

path test

list fix

fix folder creating

add install IDE

list fix

fix missing run command

Added V3 lib test + script for lib install

fix typo

add platform

updated on:

Lib testing v2

fix fqbn

test all chips

typo fix

replace blynk with Adafruit Neopixel

change blynk example

fix yml format

add action name

Add report for PR

Enable warnings report

changed position

typo

test

remove dbg level

add log level option

split command

typo

typo

add empty space

change position of cli flags

fix typo

cli compile flags not marking warning as error

Add new lib+example

install core manually

Install core by compile-sketch

fix url->path

fix typo

Install core from repo

remove external lib checkout

add lib example

update board

change platform name

change platform name

use BoardManager

update example

update example

use repo as platform

Initial commit

Update README.md

Update README.md

* fix readme links

* added JSON + lib_v2 YAML

* Update lib_v2.yml

* Update lib_v2.yml

* Update lib_v2.yml

* Update lib_v2.yml

* Update lib_v2.yml

* Update lib_v2.yml

* Update lib_v2.yml

* update JSON file

* Added json file

* Update lib.yml

* Update lib_v2.yml

* Update lib_v2.yml

* change JSON

* edit json + add version

* edit json

* update version

* fix path

* update fqbn platform name

* add empty test result file

* Add latest changes

* Update docs

* Update docs + clean YAML fifle

* Add ESPAsyncWebServer library

* Add test multiple required libs (will revert)

* Update docs + json

* Update lib.yml

* Fix typos in docs

* fix readme links

.github/workflows/lib.json

@@ -0,0 +1,56 @@
+[
+    {
+        "name": "Adafruit NeoPixel",
+        "version": "1.10.6",
+        "exclude_targets": [],
+        "sketch_path": [
+            "~/Arduino/libraries/Adafruit_NeoPixel/examples/strandtest/strandtest.ino"
+        ]
+    },
+    {
+        "name": "ArduinoBLE",
+        "exclude_targets": [
+            "esp32s2"
+        ],
+        "sketch_path": [
+            "~/Arduino/libraries/ArduinoBLE/examples/Central/Scan/Scan.ino"
+        ]
+    },
+    {
+        "name": "ESP32Servo",
+        "exclude_targets": [],
+        "sketch_path": [
+            "~/Arduino/libraries/ESP32Servo/examples/Knob/Knob.ino",
+            "~/Arduino/libraries/ESP32Servo/examples/Sweep/Sweep.ino",
+            "~/Arduino/libraries/ESP32Servo/examples/PWMExample/PWMExample.ino",
+            "~/Arduino/libraries/ESP32Servo/examples/Multiple-Servo-Example-Arduino/Multiple-Servo-Example-Arduino.ino"
+        ]
+    },
+    {
+        "source-url": "https://github.com/me-no-dev/ESPAsyncWebServer.git",
+        "required-libs": [
+            {"source-url": "https://github.com/me-no-dev/AsyncTCP.git"}
+        ],
+        "exclude_targets": [],
+        "sketch_path": [
+            "~/Arduino/libraries/ESPAsyncWebServer/examples/CaptivePortal/CaptivePortal.ino",
+            "~/Arduino/libraries/ESPAsyncWebServer/examples/ESP_AsyncFSBrowser/ESP_AsyncFSBrowser.ino",
+            "~/Arduino/libraries/ESPAsyncWebServer/examples/regex_patterns/regex_patterns.ino",
+            "~/Arduino/libraries/ESPAsyncWebServer/examples/simple_server/simple_server.ino"
+        ]
+    },
+    {
+        "name": "FastLED",
+        "exclude_targets": [],
+        "sketch_path": [
+            "~/Arduino/libraries/FastLED/examples/Blink/Blink.ino"
+        ]
+    },
+    {
+        "name": "IRremote",
+        "exclude_targets": [],
+        "sketch_path": [
+            "~/Arduino/libraries/IRremote/examples/SendDemo/SendDemo.ino"
+        ]
+    }
+]

.github/workflows/lib.yml

@@ -0,0 +1,128 @@
+name: External Libraries Test
+
+# The workflow will run on schedule and labeled pull requests
+on:
+  pull_request:
+    types: [opened, reopened, synchronize, labeled]
+
+  # Schedule weekly builds on every Sunday at 4 am
+  schedule:
+    - cron: '0 4 * * SUN' 
+
+env:
+  # It's convenient to set variables for values used multiple times in the workflow
+  SKETCHES_REPORTS_PATH: libraries-report
+  SKETCHES_REPORTS_ARTIFACT_NAME: libraries-report
+  RESULT_LIBRARY_TEST_FILE: LIBRARIES_TEST.md
+  JSON_LIBRARY_LIST_FILE: .github/workflows/lib.json
+jobs:
+  compile-sketch:
+    if: |
+      contains(github.event.pull_request.labels.*.name, 'lib_test') ||
+      (github.event_name == 'schedule' && github.repository == 'espressif/arduino-esp32')
+    runs-on: ubuntu-latest
+
+    env:
+      REPOSITORY: |
+        - source-path: '.'
+          name: "espressif:esp32"
+
+    strategy:
+      matrix:
+        target:
+          - esp32
+          - esp32s2
+          - esp32c3
+          - esp32s3
+
+        include:
+          - target: esp32
+            fqbn: espressif:esp32:esp32
+          - target: esp32s2
+            fqbn: espressif:esp32:esp32s2
+          - target: esp32c3
+            fqbn: espressif:esp32:esp32c3
+          - target: esp32s3
+            fqbn: espressif:esp32:esp32s3
+
+
+    steps:
+      # This step makes the contents of the repository available to the workflow
+      - name: Checkout repository
+        uses: actions/checkout@v3
+
+      - name: Compile sketch
+        uses: P-R-O-C-H-Y/compile-sketches@main
+        with:
+          platforms: |
+            ${{ env.REPOSITORY }}
+          target: ${{ matrix.target }}
+          fqbn: ${{ matrix.fqbn }}
+          use-json-file: true
+          json-path: ${{ env.JSON_LIBRARY_LIST_FILE }}
+          enable-deltas-report: true
+          sketches-report-path: ${{ env.SKETCHES_REPORTS_PATH }}
+          enable-warnings-report: true
+          cli-compile-flags: |
+            - --warnings="all"
+
+      - name: Upload artifact
+        uses: actions/upload-artifact@v3
+        with:
+          name: ${{ env.SKETCHES_REPORTS_ARTIFACT_NAME }}
+          path: ${{ env.SKETCHES_REPORTS_PATH }}
+
+  report-comment:
+    needs: compile-sketch  # Wait for the compile job to finish to get the data for the report
+    if: github.event_name == 'pull_request' # Only run the job when the workflow is triggered by a pull request
+    runs-on: ubuntu-latest
+    steps:
+      # This step is needed to get the size data produced by the compile jobs
+      - name: Download sketches reports artifact
+        uses: actions/download-artifact@v3
+        with:
+          name: ${{ env.SKETCHES_REPORTS_ARTIFACT_NAME }}
+          path: ${{ env.SKETCHES_REPORTS_PATH }}
+
+      - name: Report results
+        uses: P-R-O-C-H-Y/report-size-deltas@main
+        with:
+          sketches-reports-source: ${{ env.SKETCHES_REPORTS_PATH }}
+
+  report-to-file:
+    needs: compile-sketch  # Wait for the compile job to finish to get the data for the report
+    if: github.event_name == 'schedule' # Only run the job when the workflow is triggered by a schedule
+    runs-on: ubuntu-latest
+    steps:
+      # Check out repository
+      - name: Checkout repository    
+        uses: actions/checkout@v3
+
+      # This step is needed to get the size data produced by the compile jobs
+      - name: Download sketches reports artifact
+        uses: actions/download-artifact@v3
+        with:
+          name: ${{ env.SKETCHES_REPORTS_ARTIFACT_NAME }}
+          path: ${{ env.SKETCHES_REPORTS_PATH }}
+
+      - name: Report results
+        uses: P-R-O-C-H-Y/report-size-deltas@main
+        with:
+          sketches-reports-source: ${{ env.SKETCHES_REPORTS_PATH }}
+          destination-file: ${{ env.RESULT_LIBRARY_TEST_FILE }}
+
+      - name: Append file with action URL
+        uses: DamianReeves/write-file-action@master
+        with:
+          path: ${{ env.RESULT_LIBRARY_TEST_FILE }}
+          contents: |
+            / [GitHub Action Link](https://github.com/${{github.repository}}/actions/runs/${{github.run_id}})
+          write-mode: append
+
+      - name: Push to github repo
+        run: |
+          git config user.name github-actions
+          git config user.email [email protected]
+          git add ${{ env.RESULT_LIBRARY_TEST_FILE }}
+          git commit -m "Generated External Libraries Test Results"
+          git push

LIBRARIES_TEST.md

@@ -0,0 +1 @@
+Empty file

README.md

@@ -1,6 +1,7 @@
 # Arduino core for the ESP32, ESP32-S2, ESP32-S3 and ESP32-C3
 
 ![Build Status](https://github.com/espressif/arduino-esp32/workflows/ESP32%20Arduino%20CI/badge.svg) [![Documentation Status](https://readthedocs.com/projects/espressif-arduino-esp32/badge/?version=latest)](https://docs.espressif.com/projects/arduino-esp32/en/latest/?badge=latest)
+[![External Libraries Test](https://github.com/espressif/arduino-esp32/actions/workflows/lib.yml/badge.svg?branch=master&event=schedule)](https://github.com/espressif/arduino-esp32/actions/workflows/lib.yml?link=http://https://github.com/espressif/arduino-esp32/blob/master/LIBRARIES_TEST.md)
 
 ### Need help or have a question? Join the chat at [Gitter](https://gitter.im/espressif/arduino-esp32) or [open a new Discussion](https://github.com/espressif/arduino-esp32/discussions)
 
@@ -51,6 +52,11 @@ Before reporting an issue, make sure you've searched for similar one that was al
 
 Finally, if you are sure no one else had the issue, follow the **Issue template** or **Feature request template** while reporting any [new Issue](https://github.com/espressif/arduino-esp32/issues/new/choose).
 
+### External libraries compilation test
+
+We have set-up CI testing for external libraries for ESP32 Arduino core. You can check test results in the file [LIBRARIES_TEST](https://github.com/espressif/arduino-esp32/blob/master/LIBRARIES_TEST.md). 
+For more information and how to add your library to the test see [external library testing](https://docs.espressif.com/projects/arduino-esp32/en/latest/external_library_test.html) in the documentation.
+
 ### Contributing
 
 We welcome contributions to the Arduino ESP32 project!

docs/source/external_libraries_test.rst

@@ -0,0 +1,132 @@
+##########################
+External Libraries Testing
+##########################
+
+About
+-----
+
+External libraries testing is a compilation test for listed libraries on arduino-esp32 master branch. The test was made for users, so they can check which libraries are compiling without errors on which SoC.
+External libraries test is running periodically (once a week) against master branch and can also run on PR by adding a label ``lib_test``.
+The test is running on all supported ESP32 chips.
+
+.. note:: 
+  As the test is just a compilation of example, that does not guarantee that the library/sketch will run without any problems after flashing it on your device.
+
+How to Add Library to Test
+--------------------------
+
+To add a library to the CI test you need to add your library to the `lib.json`_. file located in ``./github/workflows/``.
+
+.. note:: 
+  Please add the libraries to the `lib.json`_ in alphabetical order, thanks.
+
+List of parameters:
+*******************
+
+Where the library will be installed from (use only 1 option):
+
+* ``name`` - Name of the Library in Arduino Library Manager.
+* ``source-url`` - URL to the library github repository (example: "https://github.com/Arduino-IRremote/Arduino-IRremote.git"). Use when your Library is not listed in Arduino Library Manager.
+
+Required:
+
+* ``exclude_targets`` - List of targets to be excluded from testing. Use only when the SoC dont support used peripheral.
+* ``sketch_path`` - Path / paths to the sketch / sketches to be tested.
+
+Optional:
+
+* ``version`` - Version of the library.
+* ``required-libs`` - Libraries that are necessary to be installed to compile the tested library.
+* ``destination-name`` - Folder name used for the installation of library (use only when needed).
+
+
+Example of library addition from Arduino Library Manager with ESP32-S2 excluded:
+********************************************************************************
+
+  .. code-block:: json
+
+    {
+        "name": "ArduinoBLE",
+        "exclude_targets": [
+            "esp32s2"
+        ],
+        "sketch_path": [
+            "~/Arduino/libraries/ArduinoBLE/examples/Central/Scan/Scan.ino"
+        ]
+    }
+
+Example of library addition from Github URL with required library:
+******************************************************************
+
+  .. code-block:: json
+
+    {
+        "source-url": "https://github.com/me-no-dev/ESPAsyncWebServer.git",
+        "required-libs": [
+            {"source-url": "https://github.com/me-no-dev/AsyncTCP.git"}
+        ],
+        "exclude_targets": [],
+        "sketch_path": [
+            "~/Arduino/libraries/ESPAsyncWebServer/examples/CaptivePortal/CaptivePortal.ino",
+            "~/Arduino/libraries/ESPAsyncWebServer/examples/ESP_AsyncFSBrowser/ESP_AsyncFSBrowser.ino",
+            "~/Arduino/libraries/ESPAsyncWebServer/examples/regex_patterns/regex_patterns.ino",
+            "~/Arduino/libraries/ESPAsyncWebServer/examples/simple_server/simple_server.ino"
+        ]
+    },
+
+Submit a PR
+***********
+
+* Open a PR with the changes and someone from Espressif team will add a label ``lib_test`` to the PR and CI will run the test to check, if the addition is fine and the library/example is compiling.
+
+* After merging your PR, the next scheduled test will test your library and add the results to the `LIBRARIES_TEST.md`_.
+
+Test Results
+------------
+
+Icons meaning
+*************
+
+There is a number before each icon, which means the number of examples which were compiled with that result.
+
+* |success| - Compilation was successful.
+
+* |warning| - Compilation was successful, but some warnings occurs.
+
+* |fail| - Compilation failed.
+
+* ``N/A`` - Not tested (target is in exclude_targets list).
+
+Scheduled test result
+*********************
+
+You can check the results in `LIBRARIES_TEST.md`_.
+
+The results file example:
+
+.. image:: _static/external_library_test_schedule.png
+  :width: 600
+
+Pull Request test result
+************************
+
+If the test run on Pull Request, it will compile all libraries and sketches 2 times (before/after changes in PR) to see, if the PR is breaking/fixing libraries.
+In the table the results are in order ``BEFORE -> AFTER``.
+
+.. image:: _static/external_library_test_pr.png
+  :width: 600
+
+.. |success| image:: _static/green_checkmark.png
+   :height: 2ex
+   :class: no-scaled-link
+
+.. |warning| image:: _static/warning.png
+   :height: 2ex
+   :class: no-scaled-link
+
+.. |fail| image:: _static/cross.png
+   :height: 2ex
+   :class: no-scaled-link
+
+.. _LIBRARIES_TEST.md: https://github.com/espressif/arduino-esp32/LIBRARIES_TEST.md
+.. _lib.json: https://github.com/espressif/arduino-esp32/.github/workflow/lib.json

docs/source/index.rst

@@ -19,3 +19,4 @@ Here you will find all the relevant information about the project.
    FAQ <faq>
    Troubleshooting <troubleshooting>
    Contributing <contributing>
+   External Libraries Testing <external_libraries_test>