Skip to content

Commit

Permalink
Merge pull request #57 from lincc-frameworks/issue/56/orbital-death-ray
Browse files Browse the repository at this point in the history
* Support for multiple connections via 'concurrent_connections' config
* Replaced FailedChunkCollector and old resume system in downloadCutout.py with manifest-based resume system
* Added hash and eq to downloadCutout Rects so they can be dictionary keys
* Pulled downloader functions into a class so class constants could be namespaced

* Removed pwd contextmanager in favor of absolute path filename representation in Rects.
  pwd context manager was not compatible with threading.Thread
* Stats system now outputs every 30s (controllable with stats_print_interval config)
* Stats system creates sensible metrics for a multithreaded system including how
  efficiently its multiple connections are being used (this allows us to tell on average how many connections are live with the server at once)
* Several termination issues have been fixed by making all non-main threads
  daemon threads. The result is that everything shuts down dirty on
  KeyboardInterrupt. This is okay because the only thing the main thread does
  on shutdown is read append-only manifest dicts, so additional download activity
  during the shutdown will be presumed to have failed and be overwritten on resume.
  • Loading branch information
mtauraso authored Sep 9, 2024
2 parents 2fb04de + f2cba57 commit 000c565
Show file tree
Hide file tree
Showing 4 changed files with 582 additions and 470 deletions.
Loading

0 comments on commit 000c565

Please sign in to comment.