RFC: Global Dependency System for the Dashboard

[slyubomirsky]

The dependency management via setup.sh files is cumbersome and unable to handle the common case of multiple experiments all requiring a single dependency. The current dashboard, for example, manually pulls in and builds TVM and the Relay AoT compiler. It would be ideal to have a principled way to deal with global dependencies like these, which would allow the dashboard to stand alone and be reused more easily.

I think one way to accomplish this would be a new dependency system that would work as follows: The dashboard would take a dependency directory, which contains subdirectories corresponding to each global dependency. These subdirectories will contain a setup.sh file that defines first-time setup to do and a register.sh that does any environmental setup needed for the dependency.

The dashboard will provide an empty folder in the configured setup folder to serve as space for each dependency to pull in whatever files they need and store that information. Experiments will list the dependencies they need in their configs and the dashboard will ensure that any dependencies listed by experiments will be set up first. Before an experiment runs, the register.sh file for each included dependency will run and the environment after running register.sh will be used for running the experiment's run.sh (and possibly analyze.sh as well). The same could be used for subsystems as well.

Another cumbersome aspect of dealing with dependencies and the existing setup.sh system is knowing when to rerun the setup because the experiment or the dependency has changed. For example, the current dashboard requires an argument to know if it should update its local TVM install. Dependencies can perhaps include a update.sh file that will determine if the setup should be rerun (e.g., checking if the latest commit has changed). Another possibility might be including a system for configuring policies in the dashboard itself, such as rerunning the setup after a specified interval.

A tough case would be multiple experiments/subsystems requiring different versions of the same dependency (e.g., different TVM commits). The simplest way to handle it would be to have "multiple" dependencies that are each fixed to the versions required; perhaps it might be worth having a way to pass version information to dependencies and build all the separate required versions automatically, but that is unclear.