Chocolatey doesn't always decompress downloads appropriately (support automatic decompression)

[pascalberger]

What You Are Seeing?

Vsix file downloaded using Install-ChocolateyVsixPackage is zipped, which results in checksum validation error.

What is Expected?

File should be downloaded as Vsix, without being packed in a Zip file.

How Did You Get This To Happen? (Steps to Reproduce)

Try installing this package.
Source code here

[ferventcoder]

I wonder if when it downloads it, it comes down as a zip ?

[ferventcoder]

Since this skips the verifier - can you provide a full output log (debug/verbose output)? Thanks!

[ferventcoder]

A link to a gist would be fine, or formatted here would also work.

[pascalberger]

@ferventcoder The downloaded file is named coderush-vs2015.vsix but is actually a Zip file containing the Vsix file.

Here is the log output.

[ferventcoder]

Ignoring the checksum issue for just a second, can it actually install?

[ferventcoder]

as a zip that is

[pascalberger]

No, it won't install (since it is a zip and not a vsix)

[ferventcoder]

That seems to be a much bigger issue than just failing the checksum.

[pascalberger]

@ferventcoder Any info I can provide additionally to help?

[ferventcoder]

https://devexpress.gallerycdn.vsassets.io - perhaps they are looking at the user agent and giving out a zip instead of a vsix? Are you seeing this same issue with visx files coming from the Microsoft gallery?

[pascalberger]

This is actually the Microsoft gallery (they switched recently to the Visual Studio Marketplace). Or did you mean with the old URL?

[ferventcoder]

Ah! I saw devexpress there and thought it was something else. I did mean the old URLs.

[pascalberger]

Mmh, installing https://chocolatey.org/packages/coderush-vs2015/16.1.8, which uses the visualstudiogallery.msdn.microsoft.com URL, seems to work.

[ferventcoder]

Perhaps we can talk to those folks to see what can be done.

[pascalberger]

Sure, I'll contact them about this

[AdmiringWorm]

@ferventcoder @pascalberger from what I could gather it looks like choco doesn't handle gzip encoding correctly.
No matter the Accept-Encoding Header you set they a gzip stream is returned (which I believe is handled automatically in powershell, but not in the choco helper function).

first time I've seen this happen though.

[ferventcoder]

So this is maybe an issue with choco then.

[AdmiringWorm]

it's just a possibility, I'm not entirely sure.
but from what I gather, it certainly looks like it is.

[AdmiringWorm]

found it
looks like there is a need to manually set the HttpWebRequest.AutomaticDecompression to System.Net.DecompressionMethods.GZip manually, but doing that may potentially break packages which require no decompression (I think).

Perhaps allowing a parameter to set the decompression method? (GZip, deflate or none)

[ferventcoder]

If the response header has GZip, it seems like it would make sense to set it then. Thoughts?

[AdmiringWorm]

true, it would make sense for that.
but I don't think that is possible without creating multiple web requests,
according to the docs it may throw the following exception

InvalidOperationException -> The object's current state does not allow this property to be set

I'm just guessing this would happen after parsing the content-encoding header.

[ferventcoder]

@AdmiringWorm We already create multiple requests. 👍

[AdmiringWorm]

ah yes, I see that now.

the auto detection won't solve this issue entirely.
unless a web browsers user-agent, or an accept-encoding: gzip is passed with the request to the marketplace, the response header content-encoding is not sent.
But that it something that can be set by the maintainer though.

[pascalberger]

@AdmiringWorm Where can I set this as the maintainer? I'm actually using the Install-ChocolateyVsixPackage CmdLet, which wrapps all the requests. Or can I pass the options through somehow?

[AdmiringWorm]

@pascalberger to set the the accept-encoding or any other headers you can use the options parameter:
ie:

$parameters = @{
  # Other parameters here

  Options = @{
    Headers = @{
      'Accept-Encoding' = 'gzip'
      'User-Agent' = 'whatever you want'
      # Other headers, except Connection
    }
  }
}

However this won't solve anything yet, not until the downloader is able to decompress a gzipped stream

[AdmiringWorm]

Just a small update from me regarding this,
I noticed today that ever since my last comment I had been running choco with the following added to
the `Get-WebFile.ps1' file:

$req.AutomaticDecompression = [System.Net.DecompressionMethods]::GZip -bor [System.Net.DecompressionMethods]::Deflate

That is what I had added to try installing packages from the marketplace (which then worked),
and considering I haven't even noticed anything different when upgrading/installing packages I would believe it to be safe to add.

I can open a PR for it if you're interested.

/cc @ferventcoder

[pascalberger]

@ferventcoder @AdmiringWorm Any update on this?

[ferventcoder]

I would entertain a PR that allows adding this setting. I would be a bit concerned on this chamge for wholesale of downloading things. At least at this stage, maybe it would do no harm overall.

On Tue, Feb 7, 2017 at 08:12 Pascal Berger ***@***.***> wrote: @ferventcoder <https://github.com/ferventcoder> @AdmiringWorm <https://github.com/AdmiringWorm> Any update on this? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1056 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAD4DiDOHv3skK3KNq9IXq7GNMzBrZ0qks5raHuwgaJpZM4K1XyA> .

-- Rob "Be passionate in all you do" http://codebetter.com/robreynolds http://ferventcoder.com https://twitter.com/ferventcoder

[pascalberger]

@AdmiringWorm Interested in doing the PR? Otherwise I can also give it a try.

[AdmiringWorm]

@pascalberger Feel free to give it a try if you want.

[ferventcoder]

This will be in 0.10.4