"skaffold delete --wait"

[ahmetb]

skaffold v0.8.0

If I do a skaffold delete immediately followed by a skaffold dev, I see this output, which is unexpected (note the "configured" words):

Build complete in 5.850115221s
Starting deploy...
deployment.extensions/cartservice created
service/cartservice created
deployment.extensions/checkoutservice created
service/checkoutservice created
deployment.extensions/currencyservice configured
service/currencyservice created
deployment.extensions/emailservice configured
service/emailservice created
deployment.extensions/frontend configured
service/frontend created
service/frontend-external created
deployment.extensions/paymentservice configured
service/paymentservice created
deployment.extensions/productcatalogservice configured
service/productcatalogservice created
deployment.extensions/recommendationservice configured
service/recommendationservice created
deployment.extensions/redis-cart created
service/redis-cart created
deployment.extensions/shippingservice configured
service/shippingservice created
Deploy complete in 1.505724832s
Watching for changes...

If I just did a skaffold delete I expect these resources to be deleted.

What ended up happening is that skaffold delete kicks off deletes but doesn't actually wait until deletions are complete. As a result, skaffold dev now has created these deployments:

kubectl get deployment
NAME              DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
cartservice       1         1         1            1           3m
checkoutservice   1         1         1            1           3m
redis-cart        1         1         1            1           3m

whereas if I actually run skaffold delete and wait for everything to actually get deleted, then run skaffold dev I get the full list of deployments as I expected:

kubectl get deployment
NAME                    DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
cartservice             1         1         1            1           23s
checkoutservice         1         1         1            1           23s
currencyservice         1         1         1            1           23s
emailservice            1         1         1            1           23s
frontend                1         1         1            0           23s
paymentservice          1         1         1            1           23s
productcatalogservice   1         1         1            1           23s
recommendationservice   1         1         1            1           23s
redis-cart              1         1         1            1           22s
shippingservice         1         1         1            1           22s

[r2d4]

This race condition actually gets worse when we set the grace-period to be lower. We could do a skaffold delete --wait, #176

[ahmetb]

Argh I assumed --grace-period would do a forced kill much faster. We can roll that back.

[r2d4]

We might be able to tackle it from the apply side and make sure that we're aren't trying to apply anything thats already in a Terminating state. Thats kind of unintuitive behavior from kubectl anyways.

[balopat]

I think this could be interesting for CI/CD use cases. @tejal29 what do you think?

[tstromberg]

Thank you for the idea!

I'm closing this issue as it's been open a while, and no one has yet stated an interest in addressing it. If you feels strongly about this issue, feel free to comment here or re-raise an issue referencing this one.