feat(akamai): unzip user content

[guilhem]

This pull request introduces functionality to handle compressed userdata in the Linode Metadata Service and includes the implementation of a utility function to decompress gzip data if needed. The most important changes include modifications to the fetchConfig function and the addition of a new utility function for gzip decompression.

Enhancements to userdata handling:

• internal/providers/akamai/akamai.go: Updated the fetchConfig function to decompress gzip data if needed before parsing the configuration.

New utility function:

• internal/providers/util/unzip.go: Added a new utility function GunzipIfNeeded to handle gzip decompression, including a helper function hasGzipMagicNumber to check for gzip magic numbers.

[tormath1]

This looks good to me from a functional aspect. It has been tested successfully with Flatcar with both configuration (gzipped and not gzipped):

$ linode-cli linodes create \
    --metadata.user_data "$(cat config.json | gzip | base64 -w0)" \
    --label flatcar \
    ...
$ linode-cli linodes create \
    --metadata.user_data "$(cat config.json | base64 -w0)" \
    --label flatcar \
    ...

On the instance:

Flatcar Container Linux by Kinvolk developer 9999.0.0+tormath1-ignition-akamai for Akamai
core@172-236-99-29 ~ $ sudo cat /var/run/ignition.json
{"ignition":{"config":{"replace":{"verification":{}}},"proxy":{},"security":{"tls":{}},"timeouts":{},"version":"3.5.0-experimental"},"kernelArguments...

• Tested patch here: sys-apps/ignition: pull akamai patch flatcar/scripts#2307
• Dev image available here: http://bincache.flatcar-linux.net/images/amd64/9999.0.0+tormath1-ignition-akamai/flatcar_production_akamai_image.bin.gz (instructions: https://www.flatcar.org/docs/latest/installing/community-platforms/akamai/)

[jlebon]

Doesn't the response include a Content-Encoding header? We can enhance the fetcher such that if we pass a flag via FetchOptions (or e.g. set Compression to a special value), it automatically respects Content-Encoding and decompresses it. It would then probably make sense to set this flag for all providers and not just Akamai.

[tormath1]

Doesn't the response include a Content-Encoding header? We can enhance the fetcher such that if we pass a flag via FetchOptions (or e.g. set Compression to a special value), it automatically respects Content-Encoding and decompresses it. It would then probably make sense to set this flag for all providers and not just Akamai.

At the moment it does not seem that Akamai is sending a Content-Encoding header:

core@172-233-211-243 ~ $ curl --silent -H "Metadata-Token: $TOKEN" http://169.254.169.254/v1/user-data | base64 -d | gzip -c -d -
{"ignition":{"version":"3.3.0"},"passwd":{"users":[{"name":"core","sshAuthorizedKeys":["ssh-rsa ...
core@172-233-211-243 ~ $ curl --head -H "Metadata-Token: $TOKEN" http://169.254.169.254/v1/user-data
HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 936
Content-Type: text/html; charset=utf-8
Date: Tue, 17 Sep 2024 08:02:27 GMT
Retry-After: 1
Server: nginx/1.18.0
X-Ratelimit-Limit: 10
X-Ratelimit-Remaining: 9
X-Ratelimit-Reset: 1726560149

@guilhem do you think that's something Akamai could consider?

It would then probably make sense to set this flag for all providers and not just Akamai.

That's discussed here I think: #1933 and from what I understand it's not the first time this topic is raised on Ignition. I'm just concerned about relying entirely on this Content-Encoding header - as it's up to each provider to respect this approach...

[guilhem]

I will speak to the metadata team and ask about this header :)

[guilhem]

So the header will not be added soon.
Another problem with this header is that we are embedding gzip in base64, so content is more text than gzip.

What can we do to have this PR merged?
Can we see a work on fetcher for another PR?

[guilhem]

@jlebon can we have a review? :)

[prestist]

Overall this looks great! Thank you for contributing and thank you for testing @tormath1 ❤️

Functionally it lgtm I have some small nits that I dont feel very strong about but would like to at least bring them up.

And should we add a comment about #1934 (comment) and link an rfe or somthing else

[prestist]

Hmm, this is a little bit ambiguous, small nit. Maybe we can call it something like gzipSigniture or add a comment to explain the variable.

Preferably the former, and if we do change the name I think it would clean up some of the helper functions.

[guilhem]

I change the way to test unzip by using a standard function: http.DetectContentType.

I also added test to be sure that it's working as expected

@prestist is it something better than with “magic” variables :)

[prestist]

@guilhem absolutely! 💯

[prestist]

@guilhem you have my approval, I just need to see green, looks like we have failing unit tests?

[guilhem]

@guilhem you have my approval, I just need to see green, looks like we have failing unit tests?

Awesome 😎
Sadly it's a failure in all PR.

[prestist]

Thank you again, and looking into whats going on with CI now. Thank you for bringing that to my attention.

Tracking bug here #1950

[prestist]

@guilhem if you rebase your pr should be good now
#1952 landed.

[guilhem]

@prestist rebase done.
I see Golang unit tests ok, so just waiting for all the other tests and it should be fine to merge.