[DOCUMENTATION] No Content-Encoding: gzip header on gzip-ed bindata #1556

xiaozhuai · 2020-07-10T21:31:47Z

Describe the bug
I use kataras/bindata to embeding some files, but iris reponse without Content-Encoding: gzip header

To Reproduce

go get -u github.com/kataras/bindata/cmd/bindata
bindata ./public/...
go build

	app.HandleDir("/static", "./public", iris.DirOptions{
		Asset:      GzipAsset,
		AssetInfo:  GzipAssetInfo,
		AssetNames: GzipAssetNames,
	})

The public dir has only one file index.html

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>NMac Mirror</title>
</head>
<body>
Hello!
</body>
</html>

Screenshots

Desktop
MacOS 10.14.6

The text was updated successfully, but these errors were encountered:

kataras · 2020-07-10T21:55:51Z

Hello @xiaozhuai, your bug report is correct however it is explicitly written in the kataras/bindata that the caller should be responsible to add the encoding headers. you can fix it with the AssetValidator option as shown below:

	app.HandleDir("/static", "./public", iris.DirOptions{
		Asset:      GzipAsset,
		AssetInfo:  GzipAssetInfo,
		AssetNames: GzipAssetNames,
		AssetValidator: func(ctx iris.Context, name string) bool {
			ctx.Header("Content-Encoding", "gzip")
			ctx.Header("Vary", "Accept-Encoding")
			return true
		},
	})

There is nothing more we can do there, the GzipAsset or Asset functions are not providing any information about the algorithm is used to compress the data, so we can't know. The kataras/bindata specifies that you must set the headers by yourself.

There is also no way to "extract" the content-encoding by reading the first "x" bytes of the data (as far as I know) so you must use the AssetValidator solution, which is proper and probably the most performant solution for this one. (a TODO for me would be to add this to the examples).

However, the upcoming Iris v12.2.0, which lives on master branch, has a solution for all major compression algorithms (gzip, deflate, brotli, snappy) not just gzip, there is an example:

iris/_examples/file-server/file-server/main.go

Lines 56 to 75 in c94da3b

    
           filesRouter.HandleDir("/", uploadDir, iris.DirOptions{ 
        
           	Compress: true, 
        
           	ShowList: true, 
        
           	// Optionally, force-send files to the client inside of showing to the browser. 
        
           	Attachments: iris.Attachments{ 
        
           		Enable: true, 
        
           		// Optionally, control data sent per second: 
        
           		Limit: 50.0 * iris.KB, 
        
           		Burst: 100 * iris.KB, 
        
           		// Change the destination name through: 
        
           		// NameFunc: func(systemName string) string {...} 
        
           	}, 
        
           	DirList: iris.DirListRich(iris.DirListRichOptions{ 
        
           		// Optionally, use a custom template for listing: 
        
           		// Tmpl: dirListRichTemplate, 
        
           		TmplName: "dirlist.html", 
        
           	}), 
        
           })

xiaozhuai · 2020-07-10T22:02:35Z

It works now! One more thing, for small file, it's better to not gzip. Gzipped size may be more greater. It's there any solution?

kataras · 2020-07-10T22:05:21Z

Yes, for small files the result file size may be larger, as official gzip documentation specifies. This is why gzip is not enabled by default into Iris, you have to select when it is worth and when it doesn't. You can use brotli or snappy instead, but these are only available on the master branch currently (go get -u github.com/kataras/iris/v12@master in your project's directory). See the example code I attached to you above and test it by yourself. Write down your results!

xiaozhuai · 2020-07-10T22:09:00Z

Yes, this is just an edge case.
I will try snappy.
And thanks for your kind help!

kataras · 2020-07-10T22:10:25Z

You are welcome @xiaozhuai , I've also changed the title and the labels to describe the issue with more details, if that's OK with you. Let me know if you find anything useful we can add on _examples!

xiaozhuai · 2020-07-10T22:16:45Z

I am a newbie of golang and iris.
I did some research on golang web framework, and iris is the one I like best!
I will see what I can do to contribute to this project.
Thanks!

kataras · 2020-07-11T18:10:03Z

That's awesome, thanks for the feedback @xiaozhuai. You will love golang and Iris as you learn more and more!

kataras · 2020-07-20T13:59:21Z

Yes, this is just an edge case.
I will try snappy.
And thanks for your kind help!

Hello @xiaozhuai again, two things:

Do NOT compress files smaller than 1.4KB (files smaller than size are still transmitted as one 1500-bytes TCP packet)
Do NOT compress image and pdf files (they are already compressed by nature)

I am working to improve the HandleDir as we speak, thinking of adding and implementing two new fields for that matter:

type DirOptions {
        // [other options...]

	// If Compress is set to true then this value
	// is compared against the file size to decide
	// if it should be eventually compressed or not.
	// A good value is 1.4*iris.KB, so smaller files are not compressed and waste CPU.
	//
	// Defaults to 0, it compress everything (when Compress is true).
	CompressMinSize int64
	// CompressImages enables compression over
	// already compressed file types too:
	// .pdf, .jpg, .jpeg, .gif, .png, .tif, .tiff (.svg and .bmp are not compressed).
	//
	// Defaults to false, keep it that way.
	CompressImages bool
}

Sounds good to you?

EDIT:

I also found a way to automatically add the Content-Encoding to gzip when using kataras/bindata.GzipAsset, so there is not need to add that AssetValidator we talk about previously. There is actually a way to know if the data are gzip-compressed: https://stackoverflow.com/a/6059342

xiaozhuai · 2020-07-20T14:06:01Z

@kataras Yes! That's awesome! I think this feature should be add to kataras/bindata, it should provided an option of min size and ignored file types, then provide a function to tell whether a file is compressed, so that iris can serve files with the right header.

kataras · 2020-07-20T14:14:18Z

Yes I was thinking about it but, if we read the first two bytes of each file to see if it's gzip-compressed or not, is still a performance cost that I'm not willing to give it unless a new option set by user. I don't know if it's a good idea, why not just keep the requirement of writing one line inside the AssetValidator when using GzipAsset instead of Asset?

Also, in bindata it's a good idea to add it as an argument option in terminal.

I am putting those two fields on Iris because they can be quite useful on go-bindata/go-bindata where data are not gzip-compressed already (it's more popular/more developers using it than kataras/bindata) too. Also, don't forget, you can use compression all over your Iris handlers, not just using HandleDir, e.g. ctx.ServeFile/Content and e.t.c.

then provide a function to tell whether a file is compressed, so that iris can serve files with the right header.

Iris doesn't know if the asset functions are came from kataras/bindata or go-bindata or anything else, it just accepts the function types, so this is not possible, indeed Iris is not even aware of kataras/bindata or go-bindata packages at all.

xiaozhuai · 2020-07-20T14:20:20Z

@kataras I have a question, whether iris cache compressed data of files in memory when use go-bindata? If the answer is yes, we can use go-bindata and abandon kataras/bindata. If so, we can provide any number of compression types. And that won't cause performance lose.

kataras · 2020-07-20T14:21:50Z

@xiaozhuai iris does not cache anything, the files served through Asset, AssetNames and AssetInfo functions are presented as go source code of []byte, they are already in-memory; compression happens on the fly, no it does not cache compressed data from go-bindata that's why I created the kataras/bindata at the first place, kataras/bindata exports the contents as gzip-compressed []byte.

However, your idea is not bad at all, we could scan all files and cache their gzip-bytes them before server ran (as we do on kataras/bindata, after all they wont change, they are embedded. <-- One problem here, the compression algorithm is decided on the request-time based on the client's Accept-Encoding, so some clients require gzip some others may snappy or br(brotli).

xiaozhuai · 2020-07-20T14:26:37Z

@kataras Yes, kataras/bindata provided compressed data so it don't need compress during runtime. But if we cache compress data in memory, then we can use go-bindata and have the same performance like kataras/bindata. And we can provided not only the gzip encoding. It's just a suggestion. : )

kataras · 2020-07-20T14:29:00Z

We can provide cached data for all supported compression algorithms when Compress == true && Asset != nil && AssetInfo !=nil && AssetNames !=nil -> Cache data through a map[alg:string].... on request -> cache[compression requested] ..... however this will result on larger memory.

kataras · 2020-07-20T14:34:49Z

OK, I will see what I can do, performance is everything here, we can do that. There will be probably options like above (for common file system, not embedded) plus ForceCompressAssets: bool or AssetCache: bool or AssetPrecache: bool or AssetCompressionCache: bool

xiaozhuai · 2020-07-20T14:40:14Z

@kataras Yes, that's a problem. Maybe we can provided an option that cache should be created on application initlized or the file was request first time, and provide a strategy to eliminate caches that not hit for a long time. Sorry for my poor English, hope that helps. : )

kataras · 2020-07-20T14:45:03Z

No, we can do that on build-time no serve-time (it will require locks and hassle while encoding ALL files without reason). There are just 4 builtin web compression algorithms (gzip, (deflate -> this is compatible with gzip as I can remember, so three caches), brotli, snappy).

Your english are fine, probably better than mines, so don't worry about it :)

xiaozhuai · 2020-07-20T14:47:39Z

@kataras Thank you very much! Thanks for your idea and sharing it to me!

kataras · 2020-07-20T14:48:44Z

Sure, I'll keep you informed about the progress @xiaozhuai. I thank YOU!

xiaozhuai · 2020-07-20T14:55:24Z

@kataras If you don’t mind, I’m willing to buy you some coffee. : )
Thank you for bring us iris.

kataras · 2020-07-20T15:12:02Z

Woww... I have no words... I am very touched 🥺 thank you @xiaozhuai this is very generous of you. I'm dedicated and loyal to Iris and all of you, thanks again

kataras · 2020-07-22T11:15:23Z

OK @xiaozhuai, it's almost ready to cache and compress any http.FileSystem (http.Dir (system files), and go-bindata). The go-bindata, on its latest release some months ago, has an -fs argument which generates an AssetFile() http.FileSystem function (Iris had its own version of it for views and files for years). As an extra bonus, I already implemented simple verbose to check how much each compression will help:

# cache, but, don't compress files of: .pdf, .jpg, .jpeg, .gif, .png, .tif, .tiff (already compressed by nature)
# compress files larger or equal than 1500 bytes
# with sample assets directory
$ go run .
/js/main.js.map (33 B)
/vendor/bootstrap/css/bootstrap-grid.css.map (158.9 KB)
gzip    (31.5 KB)
deflate (31.5 KB)
br      (23.6 KB)
snappy  (47.7 KB)
/vendor/bootstrap/js/bootstrap.min.js.map (191.5 KB)
br      (44.4 KB)
snappy  (77.0 KB)
gzip    (51.4 KB)
deflate (51.4 KB)
/css/main.css (41 B)
/vendor/bootstrap/js/bootstrap.bundle.min.js (81.1 KB)
deflate (23.3 KB)
br      (20.7 KB)
snappy  (34.8 KB)
gzip    (23.3 KB)
/vendor/bootstrap/css/bootstrap-grid.css (68.0 KB)
deflate (9.3 KB)
br      (5.6 KB)
snappy  (15.1 KB)
gzip    (9.3 KB)
/app2/app2app3/index.html (312 B)
/vendor/bootstrap/css/bootstrap.min.css.map (646.4 KB)
deflate (120.0 KB)
br      (75.1 KB)
snappy  (183.2 KB)
gzip    (120.0 KB)
/vendor/bootstrap/js/bootstrap.bundle.js.map (409.1 KB)
gzip    (100.7 KB)
deflate (100.7 KB)
br      (85.4 KB)
snappy  (157.0 KB)
/vendor/bootstrap/js/bootstrap.min.js (60.2 KB)
gzip    (16.2 KB)
deflate (16.2 KB)
br      (14.1 KB)
snappy  (23.4 KB)
/big_files/6KB.parts (6.3 KB)
gzip    (172 B)
deflate (154 B)
br      (117 B)
snappy  (451 B)
/app2/index.html (25 B)
/vendor/bootstrap/js/bootstrap.js.map (253.9 KB)
gzip    (63.8 KB)
deflate (63.7 KB)
br      (53.8 KB)
snappy  (98.9 KB)
/vendor/bootstrap/css/bootstrap.css.map (508.2 KB)
gzip    (112.7 KB)
deflate (112.7 KB)
br      (88.4 KB)
snappy  (170.6 KB)
/vendor/bootstrap/css/bootstrap-grid.min.css.map (115.8 KB)
gzip    (17.5 KB)
deflate (17.5 KB)
br      (12.2 KB)
snappy  (27.5 KB)
/app2/app2app3/dirs/text.txt (27 B)
/favicon.ico (15.1 KB)
snappy  (5.8 KB)
gzip    (4.1 KB)
deflate (4.1 KB)
br      (3.8 KB)
/vendor/bootstrap/css/5kb.css (4.7 KB)
gzip    (1.8 KB)
deflate (1.7 KB)
br      (1.5 KB)
snappy  (2.5 KB)
/vendor/bootstrap/css/bootstrap-reboot.min.css.map (32.3 KB)
gzip    (8.5 KB)
deflate (8.5 KB)
br      (7.9 KB)
snappy  (12.5 KB)
/vendor/bootstrap/css/bootstrap-reboot.css.map (77.3 KB)
gzip    (18.9 KB)
deflate (18.9 KB)
br      (16.8 KB)
snappy  (27.6 KB)
/big_files/Advanced_API_Security_OAuth_2.0_and_Beyond_2nd_edition.epub (8.9 MB)
gzip    (8.9 MB)
deflate (8.9 MB)
br      (8.8 MB)
snappy  (8.9 MB)
/vendor/bootstrap/css/bootstrap-grid.min.css (51.0 KB)
gzip    (7.8 KB)
deflate (7.8 KB)
br      (4.8 KB)
snappy  (13.1 KB)
/vendor/bootstrap/css/4kb.css (3.9 KB)
snappy  (2.2 KB)
gzip    (1.6 KB)
deflate (1.6 KB)
br      (1.4 KB)
/index.html (469 B)
/app2/mydir/text.txt (12 B)
/vendor/bootstrap/css/bootstrap.css (198.3 KB)
gzip    (31.1 KB)
deflate (31.1 KB)
br      (22.8 KB)
snappy  (48.2 KB)
/vendor/bootstrap/js/bootstrap.js (136.3 KB)
snappy  (43.5 KB)
gzip    (29.0 KB)
deflate (28.9 KB)
br      (23.7 KB)
/big_files/yarn-1.22.4.msi (1.6 MB)
deflate (1.4 MB)
br      (1.3 MB)
snappy  (1.4 MB)
gzip    (1.4 MB)
/js/main.js (87 B)
/vendor/bootstrap/css/bootstrap.min.css (160.4 KB)
br      (20.9 KB)
snappy  (42.5 KB)
gzip    (27.7 KB)
deflate (27.6 KB)
/vendor/bootstrap/js/bootstrap.bundle.min.js.map (315.3 KB)
deflate (90.0 KB)
br      (77.6 KB)
snappy  (135.2 KB)
gzip    (90.0 KB)
/vendor/bootstrap/js/bootstrap.bundle.js (229.2 KB)
br      (44.8 KB)
snappy  (80.2 KB)
gzip    (54.0 KB)
deflate (54.0 KB)
/app2/app2app3/css/main.css (38 B)
/app2/app2app3/dirs/dir2/text.txt (32 B)
/big_files/123kb.zip (126.5 KB)
gzip    (108.9 KB)
deflate (108.9 KB)
br      (36.1 KB)
snappy  (99.5 KB)
Time to complete compression and cache of [25/36] files multiple by [4] algorithms: 1.0430372s
Reduce total size from 14.4 MB to:
deflate (11.2 MB) [-28.76%]
br      (10.8 MB) [-32.72%]
snappy  (11.6 MB) [-23.59%]
gzip    (11.2 MB) [-28.75%]

Stay tuned!

xiaozhuai · 2020-07-22T11:22:08Z

@kataras brotli seems great. Waiting for your further good news!

remove EmbeddedDir, as latest go-bindata has an -fs option for file system read more at: kataras/iris#1556 (comment)

…for the fastest possible static file server Read HISTORY.md it contains a breaking change, second parameter of HandleDir should be iris.Dir(...) instead of just a string relative to: #1556 (comment)

kataras · 2020-07-24T10:08:01Z

@xiaozhuai I think we are ready with: d259324. Example:

iris/_examples/file-server/embedding-gzipped-files-into-app/main.go

Lines 7 to 41 in d259324

    
           // How to run: 
        
           // 
        
           // $ go get -u github.com/go-bindata/go-bindata/v3/go-bindata 
        
           // $ go-bindata -prefix "../embedding-files-into-app/assets/" -fs ../embedding-files-into-app/assets/... 
        
           // $ go run . 
        
           // Time to complete the compression and caching of [2/3] files: 31.9998ms 
        
           // Total size reduced from 156.6 kB to: 
        
           // br      (22.9 kB) [85.37%] 
        
           // snappy  (41.7 kB) [73.37%] 
        
           // gzip    (27.9 kB) [82.16%] 
        
           // deflate (27.9 kB) [82.19%] 
        
           var dirOptions = iris.DirOptions{ 
        
           	IndexName: "index.html", 
        
           	// The `Compress` field is ignored 
        
           	// when the file is cached (when Cache.Enable is true), 
        
           	// because the cache file has a map of pre-compressed contents for each encoding 
        
           	// that is served based on client's accept-encoding. 
        
           	Compress: true, // true or false does not matter here. 
        
           	Cache: iris.DirCacheOptions{ 
        
           		Enable:         true, 
        
           		CompressIgnore: iris.MatchImagesAssets, 
        
           		// Here, define the encodings that the cached files should be pre-compressed 
        
           		// and served based on client's needs. 
        
           		Encodings:       []string{"gzip", "deflate", "br", "snappy"}, 
        
           		CompressMinSize: 50, // files smaller than this size will NOT be compressed. 
        
           		Verbose:         1, 
        
           	}, 
        
           } 
        
           func newApp() *iris.Application { 
        
           	app := iris.New() 
        
           	app.HandleDir("/static", AssetFile(), dirOptions) 
        
           	return app 
        
           }

All examples have been updated. Please send a feedback if that's the expected result you searching for or not

Former-commit-id: 6e0708f87f6ee9c6efd92f7677595feb8910988a

…for the fastest possible static file server Read HISTORY.md it contains a breaking change, second parameter of HandleDir should be iris.Dir(...) instead of just a string relative to: #1556 (comment) Former-commit-id: 14b48a06fb3b99287dff543932be2937a64233b9

xiaozhuai · 2020-07-28T13:00:20Z

@kataras Sorry for the late reply, I am so happy that you make progress so quickly! I will try this new feature.

kataras · 2020-08-10T13:11:29Z

How it goes @xiaozhuai ?

xiaozhuai · 2020-08-10T13:15:48Z

@kataras Sorry, I am so busy these days. And haven't try this so far. I will try it tomorrow.

kataras · 2020-08-10T13:20:25Z

Sure! No worries, contact me if I can help you at any way

xiaozhuai added the 🐞 type:bug label Jul 10, 2020

xiaozhuai assigned kataras Jul 10, 2020

kataras added the 📒 type:documentation label Jul 10, 2020

kataras added this to the v12.2.0 milestone Jul 10, 2020

kataras added a commit that referenced this issue Jul 10, 2020

add doc for #1556

5732d44

kataras added 🤘 status:resolved and removed 🐞 type:bug labels Jul 10, 2020

kataras changed the title ~~[BUG] No Content-Encoding: gzip header~~ [DOCUMENTATION] No Content-Encoding: gzip header on gzip-ed bindata Jul 10, 2020

kataras added a commit to kataras/httpfs that referenced this issue Jul 22, 2020

cache (with compression) feature

5ce60c9

remove EmbeddedDir, as latest go-bindata has an -fs option for file system read more at: kataras/iris#1556 (comment)

kataras added a commit that referenced this issue Jul 26, 2020

add doc for #1556

3574816

Former-commit-id: 6e0708f87f6ee9c6efd92f7677595feb8910988a

xiaozhuai closed this as completed Oct 28, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[DOCUMENTATION] No Content-Encoding: gzip header on gzip-ed bindata #1556

[DOCUMENTATION] No Content-Encoding: gzip header on gzip-ed bindata #1556

xiaozhuai commented Jul 10, 2020

kataras commented Jul 10, 2020 •

edited

Loading

xiaozhuai commented Jul 10, 2020

kataras commented Jul 10, 2020

xiaozhuai commented Jul 10, 2020

kataras commented Jul 10, 2020

xiaozhuai commented Jul 10, 2020

kataras commented Jul 11, 2020

kataras commented Jul 20, 2020 •

edited

Loading

xiaozhuai commented Jul 20, 2020 •

edited

Loading

kataras commented Jul 20, 2020 •

edited

Loading

xiaozhuai commented Jul 20, 2020

kataras commented Jul 20, 2020 •

edited

Loading

xiaozhuai commented Jul 20, 2020

kataras commented Jul 20, 2020 •

edited

Loading

kataras commented Jul 20, 2020 •

edited

Loading

xiaozhuai commented Jul 20, 2020

kataras commented Jul 20, 2020

xiaozhuai commented Jul 20, 2020

kataras commented Jul 20, 2020

xiaozhuai commented Jul 20, 2020 •

edited

Loading

kataras commented Jul 20, 2020

kataras commented Jul 22, 2020

xiaozhuai commented Jul 22, 2020

kataras commented Jul 24, 2020

xiaozhuai commented Jul 28, 2020

kataras commented Aug 10, 2020

xiaozhuai commented Aug 10, 2020

kataras commented Aug 10, 2020

[DOCUMENTATION] No Content-Encoding: gzip header on gzip-ed bindata #1556

[DOCUMENTATION] No Content-Encoding: gzip header on gzip-ed bindata #1556

Comments

xiaozhuai commented Jul 10, 2020

kataras commented Jul 10, 2020 • edited Loading

xiaozhuai commented Jul 10, 2020

kataras commented Jul 10, 2020

xiaozhuai commented Jul 10, 2020

kataras commented Jul 10, 2020

xiaozhuai commented Jul 10, 2020

kataras commented Jul 11, 2020

kataras commented Jul 20, 2020 • edited Loading

xiaozhuai commented Jul 20, 2020 • edited Loading

kataras commented Jul 20, 2020 • edited Loading

xiaozhuai commented Jul 20, 2020

kataras commented Jul 20, 2020 • edited Loading

xiaozhuai commented Jul 20, 2020

kataras commented Jul 20, 2020 • edited Loading

kataras commented Jul 20, 2020 • edited Loading

xiaozhuai commented Jul 20, 2020

kataras commented Jul 20, 2020

xiaozhuai commented Jul 20, 2020

kataras commented Jul 20, 2020

xiaozhuai commented Jul 20, 2020 • edited Loading

kataras commented Jul 20, 2020

kataras commented Jul 22, 2020

xiaozhuai commented Jul 22, 2020

kataras commented Jul 24, 2020

xiaozhuai commented Jul 28, 2020

kataras commented Aug 10, 2020

xiaozhuai commented Aug 10, 2020

kataras commented Aug 10, 2020

kataras commented Jul 10, 2020 •

edited

Loading

kataras commented Jul 20, 2020 •

edited

Loading

xiaozhuai commented Jul 20, 2020 •

edited

Loading

kataras commented Jul 20, 2020 •

edited

Loading

kataras commented Jul 20, 2020 •

edited

Loading

kataras commented Jul 20, 2020 •

edited

Loading

kataras commented Jul 20, 2020 •

edited

Loading

xiaozhuai commented Jul 20, 2020 •

edited

Loading