Depencies of local libraries are not installed automatically

[nomis]

Configuration

Operating system: Ubuntu 18.04

PlatformIO Version (platformio --version): 4.0.1b3

Description of problem

If I have a local library (in lib/) with a library.json file, its dependencies are not downloaded and installed automatically. Dependency resolution then fails (but this warning is hidden behind the verbose setting).

Steps to Reproduce

1. Delete the .pio directory to remove any cached installations
2. Put a library in lib/ that has an external dependency that is not used by the project itself
3. Execute a build in verbose mode

Actual Results

The external dependency is "ignored" because it doesn't exist. The dependency graph shows the library without its dependency.

Expected Results

The dependency of the library is downloaded and the dependency graph works correctly.

Additional info

If the dependency has been downloaded to the .pio/ directory then dependency resolution works because the library can be found.

The projects in the test directory of nomis/mcu-uuid-log 2.0.1 will fail to build because the dependency uuid-common of local library uuid-log (specified in the library.json file) has not been automatically installed.

Running in verbose mode results in the following message:

Warning: Ignored `uuid-common` dependency for `uuid-log` library

[ivankravets]

Thanks for the great report. There is an issue for that #2573

We plan to fix it soon.

[ivankravets]

Please re-test with pio upgrade --dev

[Bascy]

Dependencies of libraries in /lib are downloaded now, that is perfect

But if the dependency is already present in the /lib folder, then it still is downloaded into .pio/libdeps

Example: Started with an empty libdeps folder, Sense_SPIFFSIniFile is a dependency of SenseFramework and already present in the /lib folder but it still gets downloaded into libdeps

[nomis]

It fixes my #2573 (comment) scenario 1.

As mentioned above it will conflict with scenario 2 from the same comment (but I don't mix both of these at the same time). I'm not sure how to specify the expected behaviour for this problem in a way that can be implemented.

The library.json file is not sufficient to identify a library in the PlatformIO registry because it doesn't include the username that the library will be prefixed with. If it's not possible to identify a local version of the library then you can't avoid downloading and using it as specified by dependencies. However it appears to override the local version based on the name on its own... not sure how that's supposed to work if there are two libraries with the same name locally or in the registry published by different users.

[ivankravets]

Thanks for the report. Please re-test with pio upgrade --dev.

[Bascy]

After pio upgrade --dev, using version 5.3.0b1

When trying to build the project with one of the dependencies downloaded in the /lib folder, it downloads that same library into .pio/libdeps folder and then I get this error:

Error: Traceback (most recent call last): File "C:\Users\Bas\.platformio\penv\lib\site-packages\platformio\__main__.py", line 108, in main cli() # pylint: disable=no-value-for-parameter File "C:\Users\Bas\.platformio\penv\lib\site-packages\click\core.py", line 1128, in __call__ return self.main(*args, **kwargs) File "C:\Users\Bas\.platformio\penv\lib\site-packages\click\core.py", line 1053, in main rv = self.invoke(ctx) File "C:\Users\Bas\.platformio\penv\lib\site-packages\platformio\commands\__init__.py", line 44, in invoke return super(PlatformioCLI, self).invoke(ctx) File "C:\Users\Bas\.platformio\penv\lib\site-packages\click\core.py", line 1659, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "C:\Users\Bas\.platformio\penv\lib\site-packages\click\core.py", line 1395, in invoke return ctx.invoke(self.callback, **ctx.params) File "C:\Users\Bas\.platformio\penv\lib\site-packages\click\core.py", line 754, in invoke return __callback(*args, **kwargs) File "C:\Users\Bas\.platformio\penv\lib\site-packages\click\decorators.py", line 26, in new_func return f(get_current_context(), *args, **kwargs) File "C:\Users\Bas\.platformio\penv\lib\site-packages\platformio\commands\run\command.py", line 144, in cli is_test_running, File "C:\Users\Bas\.platformio\penv\lib\site-packages\platformio\commands\run\command.py", line 179, in process_env result = {"env": name, "duration": time(), "succeeded": ep.process()} File "C:\Users\Bas\.platformio\penv\lib\site-packages\platformio\commands\run\processor.py", line 71, in process {"project_targets": build_targets}, File "C:\Users\Bas\.platformio\penv\lib\site-packages\platformio\package\commands\install.py", line 128, in install_project_env_dependencies _install_project_env_libraries(project_env, options), File "C:\Users\Bas\.platformio\penv\lib\site-packages\platformio\package\commands\install.py", line 225, in _install_project_env_libraries _install_project_private_library_deps(pkg, private_lm, env_lm, options) File "C:\Users\Bas\.platformio\penv\lib\site-packages\platformio\package\commands\install.py", line 242, in _install_project_private_library_deps _install_project_private_library_deps(pkg, private_lm, env_lm, options) File "C:\Users\Bas\.platformio\penv\lib\site-packages\platformio\package\commands\install.py", line 230, in _install_project_private_library_deps for dependency in private_lm.get_pkg_dependencies(private_pkg) or []: File "C:\Users\Bas\.platformio\penv\lib\site-packages\platformio\package\manager\base.py", line 296, in get_pkg_dependencies return self.load_manifest(pkg).get("dependencies") File "C:\Users\Bas\.platformio\penv\lib\site-packages\platformio\package\manager\base.py", line 176, in load_manifest if os.path.isdir(path) TypeError: _isdir: path should be string, bytes or os.PathLike, not NoneType

Alos, when adding a lib to the lib_deps property in platformio.ini which is already installed in the /lib folder, the library is downloaded into libdeps folder (and the same error occurs again)

[ivankravets]

Thanks for the report! Please re-run pio upgrade --dev.

[Bascy]

The error has been solved :) , but it is still downloading a library in lib_deps that is already present in the lib folder
This happens when the lib_deps folder has been fully cleared before building, or when the specific library that is already present in the lib folder is cleared from the lib_deps folder

[ivankravets]

Yes, the correct behavior for 3rd party dependencies. See the migration guide to the PlatformIO 6.0 https://docs.platformio.org/en/latest/core/migration.html and avoid Git and external sources.