-
Notifications
You must be signed in to change notification settings - Fork 10
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Download multiple files with N no of thread with progress bar #14
Comments
yes its possible from pypdl import Downloader
dl1 = Downloader()
dl.start('http://example.com/file1.txt',segments=10, block=false, display=false)
dl2 = Downloader()
dl.start('http://example.com/file2.txt', display=false)
# ..... this should allow for downloading multiple files simultaneously. the only caveat is that since both download shows progress bar it can flood the console since pypdl use a simple progress bar so you may need to create a custom progress bar to display both the output. you can access all the values like progress from the downloader object (dl.progress). or you can do a sequential download using loops in that case you can reuse the same downloader object. and set display to true this will provide u a progress bar. each file will be downloaded as multiple segment then combined to form the full download. for more advanced examples check the readme files |
I think it is a bad idea to create multiple instances to download multiple files. What if I need to download more than 500 files. Rather, I would prefer to have additional arguments to the
thread argument is to limit the no of parallel dowload at a time. It would be a good idea to implement this in your package. |
yes as you said it would be a bad idea when we are downloading a lot of files. its going to be hard to incorporate multiple downloads to start method, it think a better approch is to instead create a factory method with a parameter like max_instance(threads) which limits the number of instances at a time and reuses these instance if there is more number of downloads than that are max_instances. |
completed the implemation of PypdlFactory, you can check it out here: https://test.pypi.org/project/pypdl/. can you test and share your findings if there are any bugs. btw the name of class has been changed (Downloader -> Pypdl) you can check the updated doc in v1.4.0 branch |
How about allowing custom progress bars? This could be solved with the |
we can already use custom progress bars just set the display parameter to false and use the progress attribute in case of pypdl factory it is combining value from attributes of pypdl instances and producing a progress bar |
with tqdm(
total=downloader.size,
desc=f"Download {i}",
unit='B',
unit_scale=True,
unit_divisor=1024,
miniters=1,
position=i
) as bar :
previous = 0
while not downloader.completed :
bar.update(downloader.current_size - previous)
previous = downloader.current_size
time.sleep(0.1) This should do the job. I would appreciate if someone tested it. |
this feature has been added in v1.4.0 |
There library looks promising.
As the title says, is it possible to download multiple files (option to limit N threads) simultaneously with progress bar?
Would be happy to see some example in this.
The text was updated successfully, but these errors were encountered: