Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Too many blocks too quickly (too many files) #76

Open
antoinedelplace opened this issue Nov 28, 2023 · 2 comments
Open

Too many blocks too quickly (too many files) #76

antoinedelplace opened this issue Nov 28, 2023 · 2 comments
Assignees
Labels
bug Something isn't working

Comments

@antoinedelplace
Copy link

antoinedelplace commented Nov 28, 2023

It seems there is a problem if too many files are sent at the same time. Maybe a well positioned time.sleep could solve this problem.

Checking for no duplicate filenames
Preparing to upload
Could not find any csrf-token
From storm_upload_token WETRANSFER_STORM_PREFLIGHT URL is: https://storm-us-east-1.wetransfer.net/api/v2/batch/preflight
From storm_upload_token WETRANSFER_STORM_BLOCK URL is: https://storm-us-east-1.wetransfer.net/api/v2/blocks
From storm_upload_token WETRANSFER_STORM_BLOCK URL is: https://storm-us-east-1.wetransfer.net/api/v2/batch
Get transfer id eac9fe195aa5***0231128094504
Doing preflight storm
Preparing storm block upload

Traceback (most recent call last):
  File "/upload_wetransfer.py", line 18, in upload_wetransfer
    url = upload(files)
          ^^^^^^^^^^^^^
  File "/transferwee/transferwee.py", line 588, in upload
    for f, b in zip(files, blocks["data"]["blocks"]):
                           ~~~~~~^^^^^^^^
KeyError: 'data'

The request is sending this response :

>>> print(blocks)
>>> {'ok': False, 'error': {'client_action_required': False, 'error_code': 'TOO_MANY_REQUESTS', 'message': 'You are creating too many blocks too quickly, please try again after waiting for a little bit.'}}

I am trying to send 636 files at the same time.

@antoinedelplace
Copy link
Author

Note : trying to zip all the files and sending a unique file triggers another error.

#72 (comment)

@iamleot iamleot self-assigned this Nov 28, 2023
@iamleot iamleot added the bug Something isn't working label Nov 28, 2023
sharevb added a commit to sharevb/transferwee that referenced this issue Nov 29, 2023
As of issue iamleot#76, PR iamleot#77 and to accomodate issue iamleot#64, thus case of "many small" or "really big" files, chunk big files in many blocks and blocks request also in chunks if more than 100 blocks need for the transfer of all files
@sharevb
Copy link
Contributor

sharevb commented Nov 29, 2023

Adapted to #72

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants