/internal/v2/droplets returns 404 where /internal/v4/droplets does not

[x6j8x]

Thanks for submitting an issue to cloud_controller_ng. We are always trying to improve! To help us, please fill out the following template.

Issue

After updating to cf-deployment 2.9 we've noticed a lot of apps not coming back up with a "Downloading failed" error.

Digging deeper we've found out that the root cause for this is that all those app still have a /internal/v2/droplets/ url registered in their desired-lrp state in diego. This /v2/droplets returns a 404 now.

Manually restarting the app solves the problem (after a manual restart the url changes to /internal/v4/droplets url.

Was /v2/droplets removed recently? If yes, shouldn't there been some kind of coordination with diego regarding the migration of apps that weren't "touched" for a long time?

We now have 100s of apps on the platform in this weird state (we can't say if they're still important or not).

@emalm fyi

Crashing app:

"actions": [
        {
          "download": {
            "from": "http://cloud-controller-ng.service.cf.internal:9022/internal/v2/droplets/730dc597-c734-4a5d-b074-fd2c1ef707b3/506a828a78533f94d45125395e321ecc4de88d4f/download",
            "to": ".",
            "cache_key": "droplets-730dc597-c734-4a5d-b074-fd2c1ef707b3-fb55e262-e22a-49ef-b127-4678c2223d75",
            "user": "vcap",
            "checksum_algorithm": "sha1",
            "checksum_value": "506a828a78533f94d45125395e321ecc4de88d4f"
          }
        }
      ]

Restarted app:

"actions": [
        {
          "download": {
            "from": "https://cloud-controller-ng.service.cf.internal:9023/internal/v4/droplets/730dc597-c734-4a5d-b074-fd2c1ef707b3/506a828a78533f94d45125395e321ecc4de88d4f/download",
            "to": ".",
            "cache_key": "droplets-730dc597-c734-4a5d-b074-fd2c1ef707b3-2b76b02a-89a3-4da6-9a5d-53eb76c04d59",
            "user": "vcap",
            "checksum_algorithm": "sha1",
            "checksum_value": "506a828a78533f94d45125395e321ecc4de88d4f"
          }
        }
      ]

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/159457786

The labels on this github issue will be updated when the story is started.

[x6j8x]

fyi - we restarted all affected apps to mitigate the issue for our users.

It would be nice if something like this could be avoided in the future.

CC should have all the knowledge necessary to make this smoother (knows when an app was restarted/updated the last time).

[zrob]

Hi @x6j8x

Thanks for reporting this issue. This endpoint was accidentally removed as part of an effort to clean up some dead code. Interestingly this comment specifically notes that we should not have removed this endpoint.

We are going to add it back to potentially avoid this problem for other folks. We'll also update release notes with information regarding this problem.