Use go:embed for embedding files instead of vfsgen

[techknowlogick]

Feature Description

Now that we have a minimum of go1.16 set, we can use go's embed functionality to embed templates/web assets (css/js), and translations.

Screenshots

No response

[6543]

example:
https://github.com/woodpecker-ci/woodpecker/blob/ed6d3f3cea2c9ceab5dbcd22bcc25b48e7523ab2/web/web.go#L10-L11

[lafriks]

only we will have to do some restucture of code as you can not embed files that are not in the same directory or in child directory where golang code is where you are embedding it.

[lunny]

go:embed don't support ../ on path, the directory structure have to be changed.

[silverwind]

Maybe symlink before build?

[jolheiser]

We can add a small go file in those locations to grab the assets.

[silverwind]

Another thing to consider is that vfsgen provides the gzip versions of assets which we directly serve from the http server if the client indicates gzip support. I think it's crucial for performance to retain the option to serve pre-compiled gzip assets.

Edit:

See https://github.com/vearutop/statigz for one approach to solve it. I also maintain a JS module to generate .gz and .br files at https://github.com/silverwind/precompress.

[rahulsurwade08]

Hey @techknowlogick , I would love to contribute to this issue under Hacktoberfest. How can I get started?

[6543]

the main concern is that if we ose std embeded the compressed files would be decompressed and compressed on each request that indicated gzip support ...

... @techknowlogick should we close this issue?

[wxiaoguang]

There could be a comment in code for why the vfsgen is chosen, to help future developers know the background. Otherwise months later the knowledge may be lost.

[jolheiser]

As well, this is still a valid issue. silverwind posted above a potential https://github.com/vearutop/statigz that allows serving embedded files via gzip

[silverwind]

Yes, I think it's still a goal to use go:embed as it greatly simplifies the build by eliminating go generate and there are ways to get gzip to work by combining tools like precompress (to be ran after webpack) and statigz.

[silverwind]

We really need to migrate to go:embed. I think statigz isn't even needed as handling Accept-Encoding manually is rather trivial.

the main concern is that if we ose std embeded the compressed files would be decompressed and compressed on each request that indicated gzip support ...

If we have .gz and .br files in the embed, we can serve their bytes on-the-fly without ever decompressing. This will reduce both server load (it never needs to decompress) and optimize transfer size.

[lunny]

precompress

Doesn't vfsgen have done that?

[silverwind]

Yes, but only gzip. We want Brotli as well for another ~30% size reduction over gzip.

Using go:embed with all three variants (plain,gz,br) would be best for client performance, binary size will increase a bit, but there's no more runtime decompression involved as opposed to vfsgen.

[silverwind]

Are we fine with dropping support for non-bindata builds (e.g. always embed, don't support non-embedded assets in the file system, except custom files). I fail to see a real use case for non-bindata builds and it just complicates the implementation a lot.

[delvh]

Assuming make watch will still work: I'm okay with it.
I noticed that when I compile with TAGS="bindata …" make watch,
the frontend doesn't work anymore and as I only get the raw HTML and some browser alerts, but no CSS or JS…

[silverwind]

Indeed, watch mode depends on files in the file system. One solution is to have go run a reverse proxy on the assets path in dev mode, which would proxy asset requests to webpack-dev-server or similar.

[paralin]

Done in #22887