📚 Epic: TPI basic scenario

[dmpetrov]

Goal: get resources from CLI (through TF provider) without using CI/CD.

1. Basic Python scenario
   A user

• installs TPI on laptop
• specifies cloud creds
• modifies a instance type (like p2.xlarge), image (like ami-abcd1234) and region (like us-west).
• specifies data directory to sync to the machine (a dir for input data as well as result/model)
• specifies requirements.txt and/or Dockerfile (separate epic)
• copy to the instance train.py script and run it there

Results:

• new instance starts
• the data directory should contain both the input data and the model
• user sees log from terminal
• in case of errors the model is not synchronized but the user should see all the errors in a terminal
• the instance is terminated after the job is done (for success as well as failure).

2. Basic R scenario
   Similar to Python. But no requirements.txt

Details:

• The actions/commands users take should match the actions from CML scenario as much as possible.
• Supported clouds: AWS, Azure, GCP, K8S.
• This scenario should work without Git and DVC
• Only simple train.py synchronization is included. More complicated code sync in a separate issue 📚 Epic: Advance code sync #212

[DavidGOrtega]

@dmpetrov can we make this basic scenario language agnostic? Instead of requirements or Dockerfile it requires a script to be executed.

[0x2b3bfa0]

I like the idea of making it language–agnostic at the package level (i.e. requirements.txt, poetry.lock, packrat.lock, ...), but I do see the value of allowing OCI/Docker containers as an alternative to plain scripts.

[DavidGOrtega]

I like the idea of making it language–agnostic at the package level (i.e. requirements.txt, poetry.lock, packrat.lock, ...), but I do see the value of allowing OCI/Docker containers as an alternative to plain scripts.

Python biased (launching train.py) does not allow you to execute dvc cli nor other ML tools based on cli or others valuable programming languages like R, JS... But... whatever

[0x2b3bfa0]

Python biased (launching train.py) does not allow you to execute dvc cli nor other ML tools based on cli or others valuable programming languages like R, JS...

Agreed! Even if the two of the lock files I mentioned are for Python, the third one is for R, and they're just examples apropos of the epic.

My point is that, as you say, plain [shell] scripts are the right abstraction for providing multi–language support, but it would be nice to wrap all that in the deterministic and isolated environment provided by containers.

In the most basic use case, we would provide a cached container image with a full–featured environment, and users wouldn't even notice we use containers. If they wish, they could specify a custom container image, instead of a machine image; #146.

[DavidGOrtega]

Docker will be a special case of the executor. Gitlab's executor is exactly the same that Im talking about.

The goal of the executor is to run a shell script. For that gitlab has several strategies like docker where it provisions the container and runs the script inside, ssh where it connects to an external machine and runs there and of course plain Bash. executor. There is no need to go towards a specific Docker implementation when we can solve it first for a wider scenario.

Thats my two cents. But lets formalice our proposals and architectures

[dmpetrov]

Sure! It should be language agnostic. I added 2nd use case with R to emphasize on this Basic R scenario.

It might be nice to have some features like install requirements.txt. But it looks like a separate story / epic. Removing requirements.txt from the description.

[jbencook]

Are GCP and k8s hard requirements for this basic scenario?

[0x2b3bfa0]

Not sure how hard are they in the Mohs scale, 😄 but they work and don't seem to be a blocker.

[0x2b3bfa0]

Closed with #237