Code

Concurrent import file download#7521

Code

Merged

rtibbles merged 8 commits intolearningequality:release-v0.14.xlearningequality/kolibri:release-v0.14.xfrom kollivier:sonic_downloadskollivier/kolibri:sonic_downloadsCopy head branch name to clipboardOct 19, 2020

+89-62

Commits

Commits on Sep 19, 2020

• Start on concurrent file downloading.

  kollivier

  committed
  74fcf39

Commits on Sep 21, 2020

• Ensure an unhandled exception cancels remaining tasks.

  kollivier

  committed
  4e6e974

Commits on Sep 28, 2020

• Remove file-specific progress updating to prevent multiple threads from concurrently trying to update file-specific update progress.

  kollivier

  committed
  d286412

• Fix test failures by adjusting the is_cancelled mocks to match current behavior and making a couple tests operate on a single file to match pre-threaded test behavior.

  kollivier

  committed
  0f4774c

Commits on Oct 3, 2020

• Linting fixes.

  kollivier

  committed
  4ea47f5

• Black

  kollivier

  committed
  ae5cd36

Commits on Oct 18, 2020

• Add downloads to ThreadPoolExecutor in batches to keep memory usage down. Also, only open temp file when we start downloading.

  kollivier

  committed
  433b44b

• Fix broken tests now that len is called before downloading starts.

  kollivier

  committed
  5530496