Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Performing checksum comparison on downloaded databases in CLI #160

Merged
merged 3 commits into from
Aug 8, 2023

Conversation

Harisabdullah
Copy link
Contributor

No description provided.

@linear
Copy link

linear bot commented Aug 7, 2023

BE-2235 Perform checksum comparison on downloaded databases in CLI

We currently download free databases in the IPinfo CLI with a URL like https://ipinfo.io/data/free/country_asn.csv.gz and the token passed in, and when done we just close things off.

We have an endpoint that is the same as the above but with a /checksums suffix that returns the md5/sha1/sha256 checksums of that specific file that we would've downloaded:

$ curl 'https://ipinfo.io/data/free/country_asn.csv.gz/checksums?token=<TOKEN>'
{
  "checksums": {
    "md5": "73e94043bf749b93e99923749f422d3a",
    "sha1": "cfcade07c0b4ed1d99afd93b0b9b159e4c87e9a9",
    "sha256": "2734537ea6501b8462a1c6b91bbbb1d0588438f1fb61e98dc21ad443724e6791"
  }
}

We should perform a local sha256 checksum against the downloaded file and then check against this endpoint to compare the checksums, to ensure there was no error while downloading.

This is an idea from BE-2208 cc abdullah

@Harisabdullah Harisabdullah changed the title Performing checksums after downloading the file. Perform checksum comparison on downloaded databases in CLI Aug 7, 2023
@Harisabdullah Harisabdullah changed the title Perform checksum comparison on downloaded databases in CLI Performing checksum comparison on downloaded databases in CLI Aug 7, 2023
if err != nil {
return nil, err
}
defer resp.Body.Close()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Usually I put this right above the actual readall call instead of after a successful fetch, since we sometimes don't even read the body after a fetch as we may check for HTTP error codes / etc.

So:

resp, err := get()
if err != nil {

}

defer resp.Body.Close()
io.ReadAll()

ipinfo/cmd_download.go Outdated Show resolved Hide resolved
if localChecksum != checksumResponse.Checksums.SHA256 {
return errors.New("checksums do not match. File might be corrupted")
}

if err != nil {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This error was supposed to be connected to the download step but now it's hanging out here

ipinfo/cmd_download.go Show resolved Hide resolved
@UmanShahzad UmanShahzad self-requested a review August 8, 2023 09:21
@UmanShahzad UmanShahzad merged commit e31f0d1 into master Aug 8, 2023
@UmanShahzad UmanShahzad deleted the haris/BE-2235 branch August 8, 2023 15:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants