Slow upload speeds and eventual error "The write operation timed out"

[gingerbeardman]

I am seeing very odd behaviour with ia upload in that most of the time I get very slow upload speed.

My internet speed is 1MB/s upload which I only get around 1 in 4 tries with ia.
Each upload do (~400MB CD-ROM .ISO.7z file) should take a handful of minutes to upload.

Even when I do get 1.0s/MB (as reported in ia) the next upload will most likely be slow.
Uploads show as taking multiple hours.
Eventually, I will get "The write operation timed out".

I've checked and I have no other devices using my internet at the time of upload.
Speed test shows my connection is operating as fast or faster than advertised by my ISP.

It's an inconvenience that I cannot leave the process to run automatically due to these crashes/failures.

How can I debug or resolve this issue?

Setup

• internetarchive/ia 3.0.2
• Linux

Related?

• Derive queued before the last file (due to miscounting skipped files?) #288
• SSLError The write operation timed out #296

Tail end of crash output:

increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/volume1/@appstore/py3k/usr/local/lib/python3.8/site-packages/urllib3/packages/six.py", line 769, in reraise
    raise value.with_traceback(tb)
  File "/volume1/@appstore/py3k/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/volume1/@appstore/py3k/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 398, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/volume1/@appstore/py3k/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 239, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/volume1/@appstore/py3k/usr/local/lib/python3.8/http/client.py", line 1294, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/volume1/@appstore/py3k/usr/local/lib/python3.8/http/client.py", line 1340, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/volume1/@appstore/py3k/usr/local/lib/python3.8/http/client.py", line 1289, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/volume1/@appstore/py3k/usr/local/lib/python3.8/http/client.py", line 1084, in _send_output
    self.send(chunk)
  File "/volume1/@appstore/py3k/usr/local/lib/python3.8/http/client.py", line 1006, in send
    self.sock.sendall(data)
  File "/volume1/@appstore/py3k/usr/local/lib/python3.8/ssl.py", line 1204, in sendall
    v = self.send(byte_view[count:])
  File "/volume1/@appstore/py3k/usr/local/lib/python3.8/ssl.py", line 1173, in send
    return self._sslobj.write(data)
urllib3.exceptions.ProtocolError: ('Connection aborted.', timeout('The write operation timed out'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/volume1/@appstore/py3k/usr/local/bin//ia", line 33, in <module>
    sys.exit(load_entry_point('internetarchive==3.0.2', 'console_scripts', 'ia')())
  File "/volume1/@appstore/py3k/usr/local/lib/python3.8/site-packages/internetarchive/cli/ia.py", line 171, in main
    sys.exit(ia_module.main(argv, session))
  File "/volume1/@appstore/py3k/usr/local/lib/python3.8/site-packages/internetarchive/cli/ia_upload.py", line 298, in main
    r = _upload_files(item, local_file, upload_kwargs_copy, prev_identifier,
  File "/volume1/@appstore/py3k/usr/local/lib/python3.8/site-packages/internetarchive/cli/ia_upload.py", line 95, in _upload_files
    response = item.upload(files, **upload_kwargs)
  File "/volume1/@appstore/py3k/usr/local/lib/python3.8/site-packages/internetarchive/item.py", line 1190, in upload
    resp = self.upload_file(body,
  File "/volume1/@appstore/py3k/usr/local/lib/python3.8/site-packages/internetarchive/item.py", line 1006, in upload_file
    response = self.session.send(prepared_request,
  File "/volume1/@appstore/py3k/usr/local/lib/python3.8/site-packages/internetarchive/session.py", line 543, in send
    raise e
  File "/volume1/@appstore/py3k/usr/local/lib/python3.8/site-packages/internetarchive/session.py", line 540, in send
    reraise_modify(e, e.request.url, prepend=False)  # type: ignore
  File "/volume1/@appstore/py3k/usr/local/lib/python3.8/site-packages/internetarchive/session.py", line 537, in send
    r = super().send(request, **kwargs)
  File "/volume1/@appstore/py3k/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 701, in send 
    r = adapter.send(request, **kwargs)
  File "/volume1/@appstore/py3k/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 547, in send 
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: (ProtocolError('Connection aborted.', timeout('The write operation timed out')), 'https://s3.us.archive.org/macpeople-1999-07-01/MACPEOPLE-1999-07-01.ISO.7z')
 uploading MACPEOPLE-1999-07-01.ISO.7z:  51%|█████     | 205/401 [2:49:38<2:42:11, 49.65s/MiB]

[gingerbeardman]

I did a traceroute and saw things were being held up at my ISP servers. I wondered if they were penalising me for using one of their old modems. So I switched to the most recent one they gave me and everything is uploading full speed with no slowing down.

[gingerbeardman]

Even with my new router the timeout still happens, but it takes a lot longer to do so.

Will reopen if I can reproduce.