investigate how to speed up prebuilds on gitpod repo

[akosyakov]

Bug description

Right now there is a suspicion that gradle projects and Intellij indexing of them takes a lot of time. We should confirm whether it is true by profiling prebuilds somehow and understand why incremental prebuilds simply does not reuse dependencies and caches from last prebuild. We change these projects very rare.

Steps to reproduce

Run a prebuild on this repo.

Workspace affected

No response

Expected behavior

No response

Example repository

No response

Anything else?

No response

[felladrin]

I've noticed components/ide/jetbrains/gateway-plugin:publish is being triggered every time due to this publishToJBMarketplace: true in WORKSPACE.yaml. (Take this Werft build log as example)

Using GitHub Code Search you can see that publishToJBMarketplace is aways true which will cause Gateway Plugin to always rebuild:

• https://cs.github.com/gitpod-io/gitpod?q=jbMarketplacePublishTrigger
• https://cs.github.com/gitpod-io/gitpod?q=publishToJBMarketplace

[Explosion-Scratch]

Yeah prebuilds are taking a really long time for me (+10min). It would be nice to simply not wait for the prebuild but instead see the progress of it in the editor via terminal

[akosyakov]

@felladrin Is it the same effect in prebuilds? We cab split build and publish components, that only second depends on this flag.

[mustard-mh]

As you can see in this prebuild job(incremental) https://hw-gradle-research.preview.gitpod-dev.com/projects/gitpod/c4d70b3b-fc76-4df4-8312-b2eed367b452

Time cost in incremental prebuild tasks (download/upload from/to GCP etc not included)

Task	StartTime	EndTime	Cost
gradle	Tue 10 May 2022 03:43:35 PM UTC	Tue 10 May 2022 03:45:04 PM UTC	1:30
yarn & build	Tue 10 May 2022 03:43:16 PM UTC	Tue 10 May 2022 03:48:07 PM UTC	5:00
go	Tue 10 May 2022 03:44:44 PM UTC	Tue 10 May 2022 03:44:57 PM UTC	0:15

incremental prebuilds works well in gradle, but not yarn & build

see also incremental-prebuilds-result-log.zip

you need to search with words below to see the real job after pre-prebuild (base) loaded

♻️ Re-running task as an incremental workspace prebuild

[mustard-mh]

incremental prebuilds will download pre-prebuild (base) result first, and this step will cost a lot of time too => prebuild in US cluster but base storage in EU

[mustard-mh]

result of prebuild locate region and count of Gitpod project since May 01 in prod DB

region	count
us43xl	327
us44	2
us44xl	303

They all in us cluster

---

The road will be like this

1. Trigger incremental prebuild in US cluster
2. Workspace download base (maybe larger than 7G) which storage in GCP EU
3. After prebuild, upload result (maybe larger than 7G) to EU ( in a US cluster workspace 😅
4. Start a workspace, to make user access in IDE, which will download prebuild result (maybe larger than 7G) in EU to US cluster workspace

So in this back and forth above we cost a lot of time in uploading and downloading

[mustard-mh]

How to improve it?

• we need node_modules for code intelligent, yarn build can only run in component like gitpod-protocol. component like dashboard don't need to build first
• gradle should not be inside prebuild, it will gen folder about 7G, and every start will download it, and every stop will store it, it cost a lot.
• gradle can be gen after ws start, it'll only cost around 5 mins for all.

improve improvements above can reduce size of prebuild, so it can reduce the time for back and forth in download and upload base prebuild to reduce time of incremental prebuilds.

[mustard-mh]

incremental prebuilds got some conditions, you can check them out here

gitpod/components/server/ee/src/workspace/workspace-factory.ts

Line 194 in 81a79af

private async isGoodBaseforIncrementalPrebuild(

[mustard-mh]

This result #9874 (comment) maybe caused by the trigger of these prebuilds - Github, is in US🙈 (commit push event)

[felladrin]

• gradle should not be inside prebuild, it will gen folder about 7G, and every start will download it, and every stop will store it, it cost a lot.
• gradle can be gen after ws start, it'll only cost around 5 mins for all.

About prebuild sizes, I agree with @mustard-mh. We could transform this init into command:

gitpod/.gitpod.yml

Lines 38 to 41 in 45f11d6

	- name: Java
	init: |
	leeway exec --package components/supervisor-api/java:lib --package components/gitpod-protocol/java:lib -- ./gradlew --build-cache build
	leeway exec --package components/ide/jetbrains/backend-plugin:plugin --package components/ide/jetbrains/gateway-plugin:publish --parallel -- ./gradlew --build-cache buildPlugin

[akosyakov]

@felladrin It will imply that we need to wait 7 mins to work on backend plugin each time? 5 mins full gralde build + 2 mins of intellij indexing? I guess we indeed need to move to another rep and enable prebuilds there. Although moving to another repo does not solve the problem of long download/upload times? Pulling 10G should still be faster than 7mins? Let's move to command for now, and start looking in new setup for JB development or a discussion about running prebuild in the same cluster where they are stored?

[akosyakov]

we need node_modules for code intelligent, yarn build can only run in component like gitpod-protocol. component like dashboard don't need to build first

@mustard-mh I did not entirely understand this point. But please file a new issue for that and add to webapp inbox. cc @jankeromnes @geropl

[csweichel]

@felladrin It will imply that we need to wait 7 mins to work on backend plugin each time? 5 mins full gralde build + 2 mins of intellij indexing? I guess we indeed need to move to another rep and enable prebuilds there. Although moving to another repo does not solve the problem of long download/upload times? Pulling 10G should still be faster than 7mins? Let's move to command for now, and start looking in new setup for JB development or a discussion about running prebuild in the same cluster where they are stored?

The trade-off that we're looking at today is: "spend 10 minutes for everyone outside of the EU region to download the prebuild + egress cost" vs "spend 7 minutes for anyone who wants to work on JB, without the egress cost".

Hence, removing the gradle command for now makes 💯 sense. Moving to another repo - as you pointed out - does not solve the issue.
This prebuild performance problem will hopefully be solved with the incoming PVC work. PVC prebuilds are still a few weeks out, but we're not that far away. And, this will be a great test-case for their performance.

That's to say, remove gradle now, wait for PVC and put it back in then.

[akosyakov]

ok @mustard-mh could you make a PR about moving command and we pause on it for now

[mustard-mh]

moving to command doesn't seem to be enough, because it would make everyone who opens the gitpod repo generate 6GB of gradle cache, which is also costly to store on GS, and we advocate the concept of workspace use it up, so a lot of useless storage would be wasted, so I think we So I think we can add a command like read -p "Press enter to continue" to make the gradle not run by default, and the command will be activated explicitly only after we press the key, This is because Jetbrains developers make up a small percentage of the Gitpoder WDYT @akosyakov

[akosyakov]

@mustard-mh Could we introduce an env variable to control it? I can enable it for me always with settings then.

[mustard-mh]

@akosyakov That's fine

[akosyakov]

Let's keep this issue opened and mark tech debt till the situation with prebuild is storage is not improved. Then we should put everything back and try again. We can unassign and remove from IDE project for now though.

[felladrin]

Now that #9927 has been merged, I've noticed that even if I don't press enter to start the Java task, I still have intellisense for kotlin files in backend-plugin and gateway-plugin when I open Gitpod Repo on IntelliJ IDEA. 🤔 Makes me wonder if we actually need to run that task.

• Move java task as a command #9927

[akosyakov]

@felladrin How fast is your dev loop when you try changes? gradle build is not relevant for IntelliJ, but for testing

[mustard-mh]

related #6650

[felladrin]

@felladrin How fast is your dev loop when you try changes?

For backend-plugin I open the repo on Gitpod IntelliJ IDEA and run launch-dev-server.sh and manually restart the shell script after changing the code.

For the gateway-plugin I use local IntellJ IDEA, so it's always relying on what has been previously cached here by ./gradlew runIde.

(Not sure if this answers the question. But I can say it's already faster to develop JetBrains plugins, compared to 3 months ago.)

gradle build is not relevant for IntelliJ, but for testing

Can you tell more about it? Is it about ./gradlew check?

[akosyakov]

No, I mean that ./gradlew runIde will run build and it was already executed then you don't need to wait for it.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.