Experiments table not updating while experiments are running #4528
Comments
|
I suspect that this is related to iterative/dvc#9860 as we use the metrics/params file paths provided by There could be some other error though so if you could please take a look at the console in VS Code's developer tools and share any errors that seem to be related to DVC that would be very helpful. You can access the console like this: 
Thanks. |
mattseddon
added
A: experiments
|
There are some errors relating to The experiment 
My current |
|
Is it also expected that the "DVC: Refresh Experiments" VS Code Extension command does not work? 
(I don't know what it does, just thought I'd try and see if it makes a difference) |
|
Following the suggestion by @julieg18 to upgrade DVC to see if the other issue persists, I updated DVC to 3.15.2. Not sure if that did anything or something else, but the Experiments table just updated. I see in the "DVC" output window: |
|
This issue persists still with DVC 3.15. Could it be that |
|
I could use "DVC: Reset Persisted State and Reload Window" from the Command Palette to refresh the table, but this takes long as it has to regenerate a lot of cached data. Would it make sense to have a "Refresh" button at the top of Experiments (and Plots) editor tabs? Similar to the"Refresh Explorer" button in the file explorer? |
|
@aschuh-hf is this still happening? |
|
@mattseddon Yes, this is still happening. I currently have an experiment running and the table does not update. My workaround is to use the "+" and/or "-" buttons to include more or fewer commits because this triggers a refresh of the cashed What events would trigger the update? Could it have anything to do with my use of |
The extension uses a file watcher to call Where watched files are all of the params and metrics files shown for the workspace in To check if the file system watchers are working you can save (without making any updates) either your If you are not writing metrics to files in the workspace at all then that would be an issue. |
|
Thanks, that is useful. My workspace is a (though I'm actually using the VS Code Extension to trigger these commands from the "Experiments" tab) When I navigate to the respective experiment folder in the temp folder of the running experiment and save the However, when I do the same with the |
|
What keys are shown underneath the e.g is the Are you using a mono-repo and do you have the mono-repo open at the root? |
I'm in a mono-repo and have the DVC project (path where
These are the paths that I expect. |
|
Re #4564: The They match the keys under |
|
What else are you using the |
|
It's not that I need it as I could use I would expect DVC to work either way? Whether I open a folder or use a single- or multi-folder workspace. |
|
Edit: Not relevant. |
Would you be able to check if events are getting created (at all) for the files in question using the instructions here: https://github.com/microsoft/vscode/wiki/File-Watcher-Issues#logging-local |
|
The previous experiment was unfortunately already done and the temp folder deleted. I enabled Trace log level and started a new experiment. I notice now that even when I save When I touch the |
|
Nevermind, I found the file watcher log outputs on the remote in the "Server" (not "Log (Remote Server)") output. This is when I now touch the Note that I had reloaded the window since my previous comment (i.e., since starting the experiment) while I was looking for the trace output of the file watcher. When I touch the |
|
Looks like this is the issue: microsoft/vscode#176327 |
|
@aschuh-hf as you can see from the above PRs I've been working on getting this fixed. The next version of DVC will contain a |
|
This should be fixed with the latest versions of DVC/DVCLive/the extension. |
|
Fantastic! Thanks for working out a solution across the different subprojects. (also glad to have the |
|
For anyone else that comes across this issue the min required versions for the fix are:
|
|
I am just realizing that this doesn't fix the issue for me, because DVCLive is not running on my local machine which is executing the queued experiment task ( Instead, my I can use the I can confirm that when I My current preference would be to use the new signal file because it is outside my workspace that I may work on while the experiments are running. It would also not be too harmful if the signal file would be changed in later DVC versions as this would just mean it won't work in the future without changing my script, but it would only be the update of the Experiments table that would break (and I can link to this issue from the code for documentation on how to then resolve it). What would you suggest? |
Seems like you have all of the information required to make a decision. I don't have anything to add. |
The metrics in the Experiments table are not following the results of multiple parallel experiment runs even though
dvc exp showdoes show the different updated metrics. The Experiments table mainly contains the original base commit metrics.This issue may be because it seems that the extension does not call
dvc exp show. Instead, I see lots ofdvc stage listcommands in the "DVC" output window.There are plenty more of these; some failed with a validation of the
dvc.lockfile, e.g.,The latest commands shown at the moment are:
But
dvc exp showdoes not appear even though this is after several hours of these experiments running.The text was updated successfully, but these errors were encountered: