Shallow dependencies

[rafaeltab]

Repro: https://github.com/rafaeltab/multi-service-repro-turbo

Goals

1. Increase cache hits when changes in dependencies of dependencies don't always mean a change in the current workspace
2. Reduce frequency of releases without any changes
3. Reduce docker image sizes

Non-goals

1. Ordering without dependency
2. Depending on part of a workspace

Background

I ran into an issue where when building microservices a change in one microservice, would cause a rebuild in many other microservices, despite the change only needing to impact the microservice it occured in.

This is caused by the fact that I generate a swagger.json file based on every API, and then depend on that swagger.json file in an SDK package, which is then used by other microservices.

flowchart LR
   service_a_infra:build --> service_a_api:build
   service_a_api:build --> service_a_sdk:generate_sdk
   service_a_sdk:generate_sdk --> service_a_sdk:build
   service_a_sdk:build --> service_b_api:build

Loading

In reality, the change in the service_a_infra workspace, only really needed to cause the service_a_api:build, and service_a_sdk:generate_sdk tasks to run (as long as the swagger.json file does not change). After the service_a_sdk:generate_sdk is run, the generated sdk would be the same as a previous run, and the service_a_sdk:build task can have the same hash:

• the inputs of the task, the generated sdk
• the changed hash of the service_a_sdk:generate_sdk task is ignored

flowchart LR
   service_a_infra:build --> service_a_api:build
   service_a_api:build --> service_a_sdk:generate_sdk

Loading

Important

When the input of the service-a-sdk:build task is changed as result of the service-a-sdk:generate_sdk task, the service-a-sdk:build task does have to be rerun. This also means that when service-a-sdk:build is run, service-a-sdk:generate_sdk has to be run first (the hash of the service-a-sdk:build is only reliable after the service-a-sdk:generate_sdk task is run).

This feature is only related to the hash of the task, ordering is not changed

Example using more services, with only a change in one package service_a_infra

flowchart TD
   service_a_infra:build --> service_a_api:build
   service_a_api:build --> service_a_sdk:generate_sdk
   service_a_sdk:generate_sdk --> service_a_sdk:build
   service_a_sdk:build --> service_b_api:build
   service_b_api:build --> service_b_sdk:generate_sdk
   service_b_sdk:generate_sdk --> service_b_sdk:build
   service_b_sdk:build --> service_c_api:build
   service_c_api:build --> service_c_sdk:generate_sdk
   service_c_sdk:generate_sdk --> service_d_sdk:build
   service_d_sdk:build --> service_e_api:build
   service_e_api:build --> service_e_sdk:generate_sdk
   service_e_sdk:generate_sdk --> service_e_sdk:build

Loading

You can see how the one change goes down the list of 5 services, while only one api really needed to be rebuilt, and one sdk regenerated.

flowchart TD
   service_a_infra:build --> service_a_api:build
   service_a_api:build --> service_a_sdk:generate_sdk

Loading

Proposal

My suggested solution to this is to add shallowDependsOn, a shallow dependency only includes the input files, and environment variables in the hash but not the dependencies of the task. An example configuration for the services described in the background section could look like this:

{
  "tasks": {
    "service-a-sdk#build": {
      "shallowDependsOn": ["generate_sdk"],
      "inputs": ["sdk/**/*.ts"]
    },
    "service-b-sdk#build": {
      "shallowDependsOn": ["generate_sdk"],
      "inputs": ["sdk/**/*.ts"]
    },
    "service-c-sdk#build": {
      "shallowDependsOn": ["generate_sdk"],
      "inputs": ["sdk/**/*.ts"]
    },
    "service-d-sdk#build": {
      "shallowDependsOn": ["generate_sdk"],
      "inputs": ["sdk/**/*.ts"]
    },
    "service-e-sdk#build": {
      "shallowDependsOn": ["generate_sdk"],
      "inputs": ["sdk/**/*.ts"]
    },
    "generate_sdk": {
      "dependsOn": ["^build"],
      "input": ["swagger.json"]
    },
    "build": {
      "dependsOn": ["^build"],
    }
  }
}

The tasks before this feature would look like this:

{
  "tasks": {
    "generate_sdk": {
      "dependsOn": ["^build"],
      "input": ["swagger.json"]
    },
    "build": {
      "dependsOn": ["^build", "generate_sdk"]
    }
  }
}

I'm not completely sure I managed to explain this well, I tried my best, but feel free to ask questions if it's unclear.

[anthonyshew]

Could --only be useful in this case? It seems like you could write turbo run build generate-sdk --filter=service_a_* --only and that would omit the dependent tasks/packages, only doing what is specifically written in your turbo invocation. Curious if that fits your needs.

[rafaeltab]

I'm working on a repro with nestsjs for the services, let's see if this works :D

[rafaeltab]

I added a repro https://github.com/rafaeltab/multi-service-repro-turbo

While --only with the --filter=service_a_* would reduce the amount of builds locally, the problem is bigger in pipelines.
A rebuild of an API also means a redeploy, so you would be redeploying your services, many times, without reason.
A rebuild also takes time, so this costs 250% of the build time on average actually necessary for (almost) every change made in pipelines. (given the repro)

Besides, explaing to my team lead that every developer on the team always has to run builds with --filter and --only is not something I would want to be tasked with :).

My math for the 250% to make it more clear:
a change in service d only affects d, so that is 100% of what is necessary, a change in a rebuilds everything so 400%,
(400-100)/2=250%
So you could also say 150% extra on average.