Change url download util to use requests

[AbhiramE]

Description

The change changes the url download utility from urlretrieve library to requests. With this change I also get rid of the Tqdm utility class and move to using Tqdm inline.

Requests is a more popular library than urlretrieve.

Related Issues

Checklist:

• [ x ] I have followed the contribution guidelines and code style for this project.
• I have added tests covering my contributions.
• I have updated the documentation accordingly.
  There are tests already covering this use case in the repo. I believe this also requires no update to the documentation.

[miguelgfierro]

Some evidences here: https://stackoverflow.com/questions/2018026/what-are-the-differences-between-the-urllib-urllib2-and-requests-module

This code had been added to the NLP repo

[gramhagen]

if you want to round up you will need to avoid integer division

math.ceil(total_size / block_size)

[AbhiramE]

Do you mean doing this without the math package? Like this, (total_size+block_size-1) / block_size

…

On Thu, Jun 6, 2019, 9:17 AM Scott Graham ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In reco_utils/dataset/download_utils.py <#813 (comment)> : > Returns: str: File path of the file downloaded. """ if filename is None: filename = url.split("/")[-1] filepath = os.path.join(work_directory, filename) if not os.path.exists(filepath): - with TqdmUpTo(unit="B", unit_scale=True) as t: - filepath, _ = urlretrieve(url, filepath, reporthook=t.update_to) + + r = requests.get(url, stream=True) + total_size = int(r.headers.get("content-length", 0)) + block_size = 1024 + num_iterables = math.ceil(total_size // block_size) if you want to round up you will need to avoid integer division math.ceil(total_size / block_size) — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#813>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACNXFNNT4R22U6YOGYY3XQLPZEE6TANCNFSM4HUJ3A5Q> .

[gramhagen]

Do you mean doing this without the math package? Like this, (total_size+block_size-1) / block_size

no, using math is fine, but the // operator does integer division which will round down (10 // 3 = 3), so the ceil is not actually doing anything. but if you do (10 / 3 = 3.333) then apply ceil you will get 4 which is what I'm assuming you want.

[gramhagen]

awesome, thanks!