Run experiments in something other than the IntegratedTerminal

[mattseddon]

We want to run experiments in something other than the IntegratedTerminal, this is because the terminal acts as a black box that we cannot retrieve information from. We do not even know when a process has finished.

We need to be able to track the process of a task and even cancel it for the new experiments functionality that we are looking to implement.

We should be able to switch to some kind of Task and use the Task Provider API. This should give us all of the functionality that we need but we should try and spike the functionality before deciding this is the best way to proceed or not. This may not be the only option available so we should have another look at the docs to see what is available.

Extra information:

Previously iterative/dvc#5758 & #220 were opened to try and address this issue but we now have extra information:

We can get the path to a Python virtual environment by running the following command (with the correct Python binary)

.env/bin/python -c 'import sys; print(sys.prefix)'
/home/roger/Projects/vscode-dvc/demo/.env

python -c 'import sys; print(sys.prefix)'
/home/roger/miniconda3/envs/myenv

We can get the correct Python binary using VS Code's Python extension's public api (this was known previously).

We can then run dvc commands like this:

PATH='/home/roger/Projects/vscode-dvc/demo/.env/bin:$PATH' dvc exp run

Note: Output channels were discussed as an option for this implementation but that is not somewhere I would generally look unless something has gone wrong with an extension.

[mattseddon]

@rogermparent can we close iterative/dvc#5758?

[mattseddon]

Initial investigations are not looking good for Tasks. Reasons:

1. It appears they can only be run via 'workbench.action.tasks.runTask' which (from my current understanding) means that we may not be able to put them into any of the context menus that we already have run experiments exposed in. This may be possible but I think we would need to implement another "hack" to get it to work.
2. Both ShellExecution and ProcessExecution are both essentially black boxes with no way of knowing how far into their process they are (i.e no onDidChange callbacks). We do get onDidStartTask, onDidStartTaskProcess, onDidEndTask, and onDidEndTaskProcess from the tasks class but nothing in between.
3. CustomExecution requires a Pseudoterminal to be created which means that we can use a stream and provide our own callbacks. This options means I will still have to go through all of the pain / complexity of setting up a Pseudoterminal whilst losing some of the functionality that we already have built out.

From these findings I'll focus on getting a Pseudoterminal instance setup and writing a stream into it.

[rogermparent]

Initial investigations are not looking good for Tasks. Reasons:

1. It appears they can only be run via 'workbench.action.tasks.runTask' which (from my current understanding) means that we may not be able to put them into any of the context menus that we already have run experiments exposed in. This may be possible but I think we would need to implement another "hack" to get it to work.

We can call any arbitrary logic in a VSCode Command, as it's just a function registered to a string.
I don't see any reason we can't just call workbench.action.tasks.runTask within a Command.

2. Both ShellExecution and ProcessExecution are both essentially black boxes with no way of knowing how far into their process they are (i.e no onDidChange callbacks). We do get onDidStartTask, onDidStartTaskProcess, onDidEndTask, and onDidEndTaskProcess from the tasks class but nothing in between.

Yeah, from my own research we'll need a CustomExecution-based Task to read its output. Task is just a nice way to show the output to the user and provide standard UI elements for cancellation.

3. CustomExecution requires a Pseudoterminal to be created which means that we can use a stream and provide our own callbacks. This options means I will still have to go through all of the pain / complexity of setting up a Pseudoterminal whilst losing some of the functionality that we already have built out.

What functionality do we lose? The current functionality only just barely works and has been in need of a big rewrite for a while.

From these findings I'll focus on getting a Pseudoterminal instance setup and writing a stream into it.

👍 The Task API isn't supposed to get us out of making a Pseudoterminal, it's just a better way to show the output. Showing output with a Task also means that we don't have any need to wire up our Pseudoterminal to take user input, which sounds like a nightmare.

[mattseddon]

One thing that I missed in my previous post was that if we add any tasks they will be exposed through the command palette under this section:

Here are a list of the current tasks in my workspace:

Here are the typescript tasks available:

Here are the tasks available in one of my Python repositories:

It is work noting that these tasks are user configured and do not ship with the Python extension (here is part of my tasks.json file):

[mattseddon]

👍 The Task API isn't supposed to get us out of making a Pseudoterminal, it's just a better way to show the output. Showing output with a Task also means that we don't have any need to wire up our Pseudoterminal to take user input, which sounds like a nightmare.

I think I can build something that acts a lot like tasks run through the Task API but that we don't have to expose through Tasks: Run Task in the command palette.

Hopefully everything will start to become apparent once I can get some code up.

[mattseddon]

closed by #286.