path.d and YARP_DATA_DIRS conflict #336

traversaro · 2014-12-19T18:10:57Z

If two different projects are "Extending the search path", as defined in http://www.yarp.it/git-master/yarp_data_dirs.html , and one is extending it using the path.d method and the other one is using the YARP_DATA_DIRS method, only the path.d one is working.
More detail will follow.

The text was updated successfully, but these errors were encountered:

paulfitz · 2014-12-19T20:26:09Z

Interesting :-). Looking forward to the details.

jgvictores · 2015-01-12T16:05:56Z

For me it's the other way around...

If I set YARP_DATA_DIRS for one project, YARP stops finding the resources for a different project that is installed (even if it is cited in /usr/local/share/yarp/config/path.d).

My objective is to mainly use YARP_DATA_DIRS, but once and a while install to see that the installation (for other people) is performed correctly. Right now, if there is a lagged behind YARP_DATA_DIRS corresponding to a different project, my installed data is not found when I install. (Workaround: removing YARP_DATA_DIRS for all projects while testing installation on any specific project).

paulfitz · 2015-01-12T16:23:05Z

@jgvictores when YARP_DATA_DIRS is not set by the user, it defaults to $XDG_DATA_DIRS + /yarp/ which is where /usr/local/share/yarp/ in the default search path comes from. If you want to keep that path rather than override it, you can do something like:

export YARP_DATA_DIRS=/my/new/path:/usr/local/share/yarp

Then yarp will try /my/new/path first, and fall back on /usr/local/share/yarp. A good way to debug this is with yarp resource:

$ yarp resource --verbose 1 --find foo.ini
...
||| checking [/usr/local/share/yarp/config/path.d] (robot path.d YARP_DATA_DIRS)
...

$ YARP_DATA_DIRS='/my/new/path' yarp resource --verbose 1 --find foo.ini
...
... /usr/local/share... is gone
...

$ YARP_DATA_DIRS='/my/new/path:/usr/local/share/yarp' yarp resource --verbose 1 --find foo.ini
...
||| checking [/usr/local/share/yarp/config/path.d] (robot path.d YARP_DATA_DIRS)
...

Does that help?

paulfitz · 2015-01-12T16:24:01Z

the painful details http://wiki.icub.org/wiki/YARP_ResourceFinder

paulfitz · 2015-01-12T16:25:00Z

btw very interested in hearing ways that the new config stuff breaks old workflows (and fixing that)

jgvictores · 2015-01-12T16:46:26Z

export YARP_DATA_DIRS=/my/new/path:/usr/local/share/yarp

That is an elegant solution that works nicely. 👍

Not closing issue because it was started by @traversaro.

lornat75 · 2015-01-12T17:51:28Z

@jgvictores: given you are using these functionalities I would like to know if you find the "path.d" option useful. We added it to give users the possibility to extend YARP's search path in installed configurations, however I later thought it could be enough to give users the possibility to populate their own data files directly in the YARP installed tree (e.g. /usr/local/share/yarp).

jgvictores · 2015-01-13T09:52:51Z

@lornat75: 1) I like extending YARP's search path in installed configurations. 2) Regarding installing an external package directly in YARP... personally, I prefer to keep YARP a bit cleaner.

In Linux, I find path.d very useful because, in its current state, it supports (1): since both YARP and the external package go to /usr/local/share/, /usr/local/share/yarp's path.d is nicely automatically set to point to the installed external package's resources.
In Windows, the current implementation forces (2). This is, either I force the external package resources to go in YARP's directory (since paths are a bit more ad-hoc, there's no /usr/local/share/ or XGD spec), or I will end up with no automatic setting of YARP's path.d to point to the installed external package's resources. While this could be solved by developing a separate windows installer that sets YARP_DATA_DIRS, in an ideal case, I would prefer something that automatically enables (1). What would your opinion be on enabling this, in terms of implementation and best practices?

traversaro · 2015-12-29T11:52:01Z

More details. yarp and icub-main installed from sources and installed in their default installation directories.
iCub_SIM works fine if YARP_DATA_DIRS is not defined, see how it finds the Sim_joints.ini configuration file:

pegua@pareto:~$ yarp resource --context simConfig --find Sim_joints.ini
||| configuring
||| no policy found
||| added context simConfig
||| finding file [Sim_joints.ini]
||| checking [/home/pegua/Sim_joints.ini] (pwd)
||| checking [/home/pegua/.config/yarp/robots/icubGazeboSim] (robot YARP_CONFIG_HOME)
||| checking [/home/pegua/.local/share/yarp/robots/icubGazeboSim] (robot YARP_DATA_HOME)
||| checking [/etc/xdg/xdg-xubuntu/yarp/robots/icubGazeboSim] (robot YARP_CONFIG_DIRS)
||| checking [/usr/share/upstart/xdg/yarp/robots/icubGazeboSim] (robot YARP_CONFIG_DIRS)
||| checking [/etc/xdg/yarp/robots/icubGazeboSim] (robot YARP_CONFIG_DIRS)
||| checking [/usr/share/xubuntu/yarp/robots/icubGazeboSim] (robot YARP_DATA_DIRS)
||| checking [/usr/share/xfce4/yarp/robots/icubGazeboSim] (robot YARP_DATA_DIRS)
||| checking [/usr/local/share/yarp/robots/icubGazeboSim] (robot YARP_DATA_DIRS)
||| checking [/usr/share/yarp/robots/icubGazeboSim] (robot YARP_DATA_DIRS)
||| checking [/usr/share/xubuntu/yarp/config/path.d] (robot path.d YARP_DATA_DIRS)
||| checking [/usr/share/xfce4/yarp/config/path.d] (robot path.d YARP_DATA_DIRS)
||| checking [/usr/local/share/yarp/config/path.d] (robot path.d YARP_DATA_DIRS)
||| found /usr/local/share/yarp/config/path.d
||| checking [/usr/local/share/iCub/robots/icubGazeboSim] (robot yarp.d)
||| checking [/home/pegua/.config/yarp/contexts/simConfig] (context YARP_CONFIG_HOME)
||| checking [/home/pegua/.local/share/yarp/contexts/simConfig] (context YARP_DATA_HOME)
||| found /home/pegua/.local/share/yarp/contexts/simConfig
||| checking [/etc/xdg/xdg-xubuntu/yarp/contexts/simConfig] (context YARP_CONFIG_DIRS)
||| checking [/usr/share/upstart/xdg/yarp/contexts/simConfig] (context YARP_CONFIG_DIRS)
||| checking [/etc/xdg/yarp/contexts/simConfig] (context YARP_CONFIG_DIRS)
||| checking [/usr/share/xubuntu/yarp/contexts/simConfig] (context YARP_DATA_DIRS)
||| checking [/usr/share/xfce4/yarp/contexts/simConfig] (context YARP_DATA_DIRS)
||| checking [/usr/local/share/yarp/contexts/simConfig] (context YARP_DATA_DIRS)
||| checking [/usr/share/yarp/contexts/simConfig] (context YARP_DATA_DIRS)
||| checking [/usr/local/share/iCub/contexts/simConfig] (context yarp.d)
||| found /usr/local/share/iCub/contexts/simConfig
||| checking [/home/pegua/.local/share/yarp/contexts/simConfig/Sim_joints.ini] (context)
||| checking [/usr/local/share/iCub/contexts/simConfig/Sim_joints.ini] (context)
||| found /usr/local/share/iCub/contexts/simConfig/Sim_joints.ini
"/usr/local/share/iCub/contexts/simConfig/Sim_joints.ini"

As soon as the YARP_DATA_DIRS environmental variable is defined, Sim_joints.ini is not found anymore:

pegua@pareto:~$ YARP_DATA_DIRS="" yarp resource --context simConfig --find Sim_joints.ini
||| configuring
||| no policy found
||| added context simConfig
||| finding file [Sim_joints.ini]
||| checking [/home/pegua/Sim_joints.ini] (pwd)
||| checking [/home/pegua/.config/yarp/robots/icubGazeboSim] (robot YARP_CONFIG_HOME)
||| checking [/home/pegua/.local/share/yarp/robots/icubGazeboSim] (robot YARP_DATA_HOME)
||| checking [/etc/xdg/xdg-xubuntu/yarp/robots/icubGazeboSim] (robot YARP_CONFIG_DIRS)
||| checking [/usr/share/upstart/xdg/yarp/robots/icubGazeboSim] (robot YARP_CONFIG_DIRS)
||| checking [/etc/xdg/yarp/robots/icubGazeboSim] (robot YARP_CONFIG_DIRS)
||| checking [/home/pegua/.config/yarp/contexts/simConfig] (context YARP_CONFIG_HOME)
||| checking [/home/pegua/.local/share/yarp/contexts/simConfig] (context YARP_DATA_HOME)
||| found /home/pegua/.local/share/yarp/contexts/simConfig
||| checking [/etc/xdg/xdg-xubuntu/yarp/contexts/simConfig] (context YARP_CONFIG_DIRS)
||| checking [/usr/share/upstart/xdg/yarp/contexts/simConfig] (context YARP_CONFIG_DIRS)
||| checking [/etc/xdg/yarp/contexts/simConfig] (context YARP_CONFIG_DIRS)
||| checking [/home/pegua/.local/share/yarp/contexts/simConfig/Sim_joints.ini] (context)
||| checking [/home/pegua/.config/yarp/Sim_joints.ini] (YARP_CONFIG_HOME)
||| checking [/home/pegua/.local/share/yarp/Sim_joints.ini] (YARP_DATA_HOME)
||| checking [/etc/xdg/xdg-xubuntu/yarp/Sim_joints.ini] (YARP_CONFIG_DIRS)
||| checking [/usr/share/upstart/xdg/yarp/Sim_joints.ini] (YARP_CONFIG_DIRS)
||| checking [/etc/xdg/yarp/Sim_joints.ini] (YARP_CONFIG_DIRS)
||| did not find Sim_joints.ini
""

traversaro · 2015-12-29T13:44:57Z

The point is in the ResourceFinder::getDataDirs() method.

If YARP_DATA_DIRS is set, the default paths used for searching configuration file (including configuration files for extending the search path using the path.d directory!) are not considered at all.

This seems to be directly contradicting the documentation, in which YARP_DATA_DIRS is described as a method to extend the search path [1], not to redefine it.

There are then two possible solutions:

Fix the code to match the documentation
Fix the documentation to match the code

I would prefer the first solution (fixing the code) to reduce the complexity in documenting the installation process of downstream projects such as the codyco-superbuild. Otherwise It will be needed to explicitly explain in the installation instructions of downstream projects that YARP_DATA_DIRS should manually contain the default dirs used by yarp (that thus need to be documented), to avoid problems such as robotology/icub-main#262 .

cc @drdanz

[1] : http://www.yarp.it/yarp_data_dirs.html#datafiles_extendsearch

lornat75 · 2016-01-05T22:04:38Z

If we decide to support path.d as a way to extend yarp's search path for resources maybe it makes sense for YARP_DATA_DIRS to redefine it.

However, I do not have strong opinions your observations look reasonable too.

traversaro · 2017-03-13T13:57:34Z

@diegoferigo was recently affected by this.

traversaro changed the title ~~path.d and YARP_DATA_DIRS~~ path.d and YARP_DATA_DIRS conflict Dec 19, 2014

jgvictores mentioned this issue Jan 15, 2015

icub-main cannot find resources on Windows:_installation_from_sources (tested on VS10) #354

Closed

traversaro mentioned this issue Dec 29, 2015

icub_SIM it does not work anymore robotology/icub-main#262

Closed

jgvictores mentioned this issue Jun 16, 2016

Images resources searched with a local path asrob-uc3m/robotDevastation#51

Closed

11 tasks

traversaro mentioned this issue Sep 19, 2019

update setup.sh for human-dynamics profile robotology/robotology-superbuild#257

Merged

traversaro mentioned this issue Feb 20, 2020

Migrate codyco-modules/src/modules/torqueBalancing/app/robots/ configuration files to some other repo robotology-legacy/codyco-modules#273

Closed

drdanz mentioned this issue Feb 6, 2021

Add CMake option to force installation of path.d-related files in yarp_configure_external_installation #2444

Closed

traversaro mentioned this issue May 25, 2021

Investigate automatic support for autocomplete integration robotology/robotology-superbuild#622

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

path.d and YARP_DATA_DIRS conflict #336

path.d and YARP_DATA_DIRS conflict #336

traversaro commented Dec 19, 2014 •

edited by drdanz

Loading

paulfitz commented Dec 19, 2014

jgvictores commented Jan 12, 2015

paulfitz commented Jan 12, 2015

paulfitz commented Jan 12, 2015

paulfitz commented Jan 12, 2015

jgvictores commented Jan 12, 2015

lornat75 commented Jan 12, 2015

jgvictores commented Jan 13, 2015

traversaro commented Dec 29, 2015

traversaro commented Dec 29, 2015

lornat75 commented Jan 5, 2016

traversaro commented Mar 13, 2017

path.d and YARP_DATA_DIRS conflict #336

path.d and YARP_DATA_DIRS conflict #336

Comments

traversaro commented Dec 19, 2014 • edited by drdanz Loading

paulfitz commented Dec 19, 2014

jgvictores commented Jan 12, 2015

paulfitz commented Jan 12, 2015

paulfitz commented Jan 12, 2015

paulfitz commented Jan 12, 2015

jgvictores commented Jan 12, 2015

lornat75 commented Jan 12, 2015

jgvictores commented Jan 13, 2015

traversaro commented Dec 29, 2015

traversaro commented Dec 29, 2015

lornat75 commented Jan 5, 2016

traversaro commented Mar 13, 2017

traversaro commented Dec 19, 2014 •

edited by drdanz

Loading