[BUG] flytesnacks Error 124 (timeout)

[mneilly-et]

Describe the bug
Following the tutorial at https://flytecookbook.readthedocs.io/en/latest/tutorial.html and running "make start" results in an "Error 124" due to the following timeout command in the Makefile:

timeout 30 sh -c "until kubectl explain deployment &> /dev/null; do sleep 1; done"

A 30 second timeout does not appears to be long enough.

$ make start                                                                                                              
Preparing dependencies                                                   
[+] Building 1.3s (11/11) FINISHED                                       
 => [internal] load build definition from Dockerfile                                                                      0.0s                     
 => => transferring dockerfile: 38B                                                                                       0.0s                     
 => [internal] load .dockerignore                                                                                         0.0s                     
 => => transferring context: 2B                                                                                           0.0s                     
 => [internal] load metadata for docker.io/library/docker:20.10.3-dind                                                    1.0s                     
 => [1/5] FROM docker.io/library/docker:20.10.3-dind@sha256:43df6c4dbc32ccb306412fcce5b1443310cf9d91507409d7258cea1c9ee7  0.0s                     
 => [internal] load build context                                                                                         0.0s                     
 => => transferring context: 39B                                                                                          0.0s                     
 => https://github.com/k3s-io/k3s/releases/download/v1.20.2%2Bk3s1/k3s                                                    0.2s                     
 => CACHED [2/5] RUN apk add --no-cache bash git make rsync tini                                                          0.0s                     
 => CACHED [3/5] ADD https://github.com/k3s-io/k3s/releases/download/v1.20.2%2Bk3s1/k3s /usr/local/bin/k3s                0.0s                     
 => CACHED [4/5] RUN chmod +x /usr/local/bin/k3s                                                                          0.0s                     
 => CACHED [5/5] COPY k3s-entrypoint.sh /usr/local/bin                                                                    0.0s                     
 => exporting to image                                                                                                    0.0s                     
 => => exporting layers                                                                                                   0.0s                     
 => => writing image sha256:834b8db9baf1ad4e898bad678986da2d1b295afcb484a1203f90004f372ea5c9                              0.0s                     
 => => naming to docker.io/library/flyte-sandbox:latest                                                                   0.0s                     
Starting sandboxed Kubernetes cluster                                                                                                              
make: *** [Makefile:59: start] Error 124   

Expected behavior
No timeout error.

Additional context
To Reproduce
Steps to reproduce the behavior:

1. pip install --pre flytekit
2. git clone [email protected]:flyteorg/flytesnacks.git flytesnacks
3. cd flytesnacks
4. make start

[kumare3]

cc @jeevb / @EngHabu ^^

[kumare3]

@mneilly-et I agree, 30 seconds here would rely on network speed as well, so that images can be downloaded within that time-period. @jeevb / @EngHabu can we raise it to much higher like 10 minutes? And in the case of failure, probably write a clear message that reasons why its taking more than 10 minutes - that, fail to start deployments - status of the deploymen?

@mneilly-et on the other hand, we are trying to optimize the deployment to have a fast startup time - thank you for bearing with us

[mneilly-et]

FYI - as a secondary issue, when the timeout occurs the container keeps running and issuing "make start" again results in a naming conflict. Maybe stopping the container on failure by default and having a variable to request not stopping if debug is needed would be good. FWIW.

[EngHabu]

@jeevb will migrate this to a newer image to avoid image build and potential timeout.. I think bumping timeout is also a good idea...

[kumare3]

@EngHabu / @jeevb what do you think is required more to close this one out?

[jeevb]

The timeouts were increased until we can start using the flyte-sandbox image for this: flyteorg/flytesnacks@1364536