Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integrate archiver to simplify download #382

Closed
wants to merge 1 commit into from

Conversation

ferhatelmas
Copy link
Contributor

@ferhatelmas ferhatelmas commented Nov 13, 2019

Right now, archiver doesn't handle security issue
but there is an existing PR to handle it.
Added walker to handle until upstream is fixed.

fixes #371

@k8s-ci-robot k8s-ci-robot added do-not-merge/invalid-commit-message Indicates that a PR should not merge because it has an invalid commit message. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Nov 13, 2019
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: ferhatelmas
To complete the pull request process, please assign ahmetb
You can assign the PR to them by writing /assign @ahmetb in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Right now, archiver doesn't handle security issue
but there is an existing PR to handle it.
Added walker to handle until upstream is fixed.
When it's updated with go modules, walker can be dropped.
@k8s-ci-robot k8s-ci-robot removed the do-not-merge/invalid-commit-message Indicates that a PR should not merge because it has an invalid commit message. label Nov 13, 2019
@codecov-io
Copy link

codecov-io commented Nov 13, 2019

Codecov Report

Merging #382 into master will decrease coverage by 0.54%.
The diff coverage is 63.41%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #382      +/-   ##
==========================================
- Coverage   57.11%   56.56%   -0.55%     
==========================================
  Files          19       19              
  Lines         914      861      -53     
==========================================
- Hits          522      487      -35     
+ Misses        339      330       -9     
+ Partials       53       44       -9
Impacted Files Coverage Δ
pkg/download/downloader.go 70.96% <63.41%> (+2.27%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b9a0e88...c95f3c8. Read the comment docs.

@corneliusweig
Copy link
Contributor

@ferhatelmas Can you link the PR you are talking about?

@ferhatelmas
Copy link
Contributor Author

sure, mholt/archiver#169

@corneliusweig
Copy link
Contributor

Hmm.. strange. The repo seems pretty active but this CVE fix is from May 12. What's wrong there?

@ferhatelmas
Copy link
Contributor Author

I guess the owner was busy with caddy 2.0 release and there is no other maintainer.

case zip.FileHeader:
return suspiciousPath(h.Name)
default:
return errors.Errorf("Unknow header type: %T", h)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also: Error strings should not be capitalized
Ref: https://github.com/golang/go/wiki/CodeReviewComments#error-strings

@ahmetb
Copy link
Member

ahmetb commented Nov 15, 2019

Can we please preserve signatures of something like extractZIP and extractTARGZ and/or write interface wrappers around the mholt/archiver package please?

I know this is not super necessary but for example see how we have wrappers around similar packages such as semver package and yaml packages.

This is a pretty critical dependency so I'd love to have an option to flip between implementations if it comes to that.

@corneliusweig
Copy link
Contributor

I'm a bit worried that the maintainer has not managed to merge a proposed fix for a CVE in half a year.

Besides, archiver just unpacks and creates symlinks:
https://github.com/mholt/archiver/blob/6041b49af61de1075782610aaee8c6c53ba73c45/zip.go#L207-L215
and
https://github.com/mholt/archiver/blob/6041b49af61de1075782610aaee8c6c53ba73c45/tar.go#L232-L245

Thus, it's a piece of cake to escape from the target directory and write to arbitrary paths (you don't even have to chain symlinks).

To me it clearly looks like archiver was built with ease in mind, but not security. It's also not possible to configure its security level, so I'm afraid we can't use archiver.

Copy link

@erain erain left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Most lgtm. It is indeed cleanup the code by using archiver lib.

case zip.FileHeader:
return suspiciousPath(h.Name)
default:
return errors.Errorf("Unknow header type: %T", h)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also: Error strings should not be capitalized
Ref: https://github.com/golang/go/wiki/CodeReviewComments#error-strings

@corneliusweig
Copy link
Contributor

corneliusweig commented Nov 15, 2019

@erain What is your opinion about the symlink handling in archiver?

@erain
Copy link

erain commented Nov 15, 2019

@corneliusweig You are talking about the security implication in #359 right?

I am not the expert on the topic. Some research showed that archiver has symlink support built-in: mholt/archiver#92

I am not sure about whether the symlink handling is ideal (e.g. only relative symlink and not go beyond the base dir) yet. But we can do more research here.

@ahmetb
Copy link
Member

ahmetb commented Nov 15, 2019

I vote -1 for using archiver. It’s not in a good state. We seem to be caring about security more. And we have open PRs like #360 which will rely on our ability to correctly and securely implement symlinks.

@erain
Copy link

erain commented Nov 15, 2019

Good to know. Sorry for missing the context on this.

@corneliusweig
Copy link
Contributor

@erain no worries at all. There is a history about escaping the unpack directory, so that is something to keep an eye out for.

@corneliusweig
Copy link
Contributor

corneliusweig commented Nov 15, 2019

/close
The current state of archiver does not meet our security requirements. Therefore we cannot integrate it into our code-base.

@k8s-ci-robot
Copy link
Contributor

@corneliusweig: Closed this PR.

In response to this:

/close
In the current state of archiver it does not meet our security requirements.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@corneliusweig
Copy link
Contributor

@ferhatelmas Thanks though for investigating and integrating into krew. Maybe there is another better alternative?

@ahmetb
Copy link
Member

ahmetb commented Nov 15, 2019

For now, I recommend we consider creating an interface, and extracting unarchive functionality to its new pkg that we can more easily test/maintain.

We can even make tar/zip methods smaller as there are a lot of commonalities during extraction (mkdir, write file, symlink...).

Let's just start with simply moving the code, and keep PRs small/reviewable.

@ferhatelmas
Copy link
Contributor Author

ferhatelmas commented Nov 16, 2019

Thanks though for investigating and integrating into krew.

No worries.

Let's just start with simply moving the code, and keep PRs small/reviewable

I will try this. Then, I will investigate another option to offload implementation.

By the way, #360 should go in first, right?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Move tar and zip extraction into a reusable package
6 participants