Logs of completed prebuilds are not loaded due to CORS issues

[AlexTugarev]

Bug description

Browser agents won't load prebuild logs from pre-signed URL like

https://storage.googleapis.com/gitpod-prod-user-000000000000000000000000000/workspaces/teal-sadf-asdf/instances/000000-11111-22222-33333/logs/0?Expires=1628177023&GoogleAccessId=prod-meta-eu01-storage%40gitpod-191109.iam.gserviceaccount.com&Signature=

CORS error from the network tab is MissingAllowOriginHeader

blocked by CORS policy: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource.

cc. @geropl, @csweichel

Steps to reproduce

Try to access logs of a prebuild after it's done.

Expected behavior

CORS settings are applied to the bucket and GCS will response with the missing headers.

Example repository

No response

Anything else?

No response

[ajhalili2006]

Does Google Cloud allows the access of files in a storage bucket as part of CORS request or do we need to do some NGINX/Caddy proxy tricks?

[csweichel]

We could set CORS config when creating a new bucket, something like here:

CORS: []gcpstorage.CORS{
	{
		Origins: []string{"*"},
		Methods: []string{"GET"},
		MaxAge:  6 * time.Hour,
	},
},

However, this comes with a set of drawbacks:

• in this naive form it would only work for new buckets
• it only works in GCP - for minio no such option exists. Does minio need this?

The other solution, route it all through the proxy is something we're already doing for workspace content download. We have a custom caddy plugin that implements the mechanism. We could do something similar here.

The main question we'd need to answer is if in this case we'd be paying for egress twice: once from the bucket to our proxy, and then from the proxy to the client (not so much of a concern for workspace content download - this doesn't happen so often). Frankly, I would not expect this to be a problem, we should check.

[AlexTugarev]

@csweichel, first I was not sure if applying these bucket attributes in general would make sense, because it all contents of /gitpod-prod-user-000000000000000000000000000/**/*. But there isn't any additional risk to that either, because once the URL is served it can be used with HTTP clients other than browsers to ignore CORS.

[csweichel]

We'd need to apply the CORS attributes to all buckets. The one you refer to only holds the image builds, but not prebuild logs.
In the future the image build logs will carry user association as well, hence be in the user buckets, too.

[AlexTugarev]

The one you refer to only holds the image builds, but not prebuild logs.

Nope, that's the log output of the workspace instance for the prebuild I'm interested in. These URLs are returned by the server now.

WDYT of patching this when we call ensureExists?

[csweichel]

The one you refer to only holds the image builds, but not prebuild logs.

Nope, that's the log output of the workspace instance for the prebuild I'm interested in. These URLs are returned by the server now.

I meant that gitpod-prod-user-000000000000000000000000000 only holds the image-build logs.

WDYT of patching this when we call ensureExists?

Do we have the same CORS issue with minio?

[AlexTugarev]

🤦🏻 oh this does exist indeed. I used that as a template. replace with gitpod-prod-user-ID then, please

[JanKoehnlein]

Sorry, but due to high load and little resources in groundwork, we cannot schedule this issue ATM.
Please try to solve it with the crew or bring it in again in a few days.

[svenefftinge]

The other solution, route it all through the proxy is something we're already doing for workspace content download. We have a custom caddy plugin that implements the mechanism. We could do something similar here.

I'm much in favor of this, as it provides us a central control and solves this for other storage providers (minio). We are talking about log files of prebuilds so I assume egress isn't a problem here?

[svenefftinge]

FWIW here's the caddy plugin for the workspace content download : https://github.com/gitpod-io/gitpod/blob/main/components/proxy/plugins/workspacedownload/workspace_download.go

[svenefftinge]

Would be good if someone (@geropl, @corneliusludmann, @aledbf maybe?) with more knowledge of caddy can take this. I assume it would be relatively easy based on the existing plugin.

[corneliusludmann]

I can have a look but I don't know much about Teams & Projects yet. Can someone give me an intro to what I need to do to be able to download prebuild logs?

[csweichel]

/team meta

[csweichel]

/schedule

[roboquat]

@csweichel: Issue scheduled in the meta team (WIP: 0)

In response to this:

/schedule

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[geropl]

The code is #5480, but needs to be tested. Was blocked by some problems with nodeAffinity of imagebuild workspaces (issue).