You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Using the current develop branch (commit f022d06), I saw some issues when trying to transfer larger files. While file transfers for smaller files worked fine, I saw issues when trying upload a 17GB and a 100GB tar archive.
First some general info:
Python version used: Python 3.10.6
All other required Python packages are in the version specified in the requirements.txt
The issue in more detail:
Trying to upload a 100GB tar archive via the Data Transfers tab
After a longer time, the client crashes and the following message is displayed:
['/home/jelte/peregrine_mount/100GB.tar'] 100000010240
100000010240 0
[2023-02-15 13:28:15,016] {IrodsConnector.py:967} INFO - iRODS UPLOAD: /home/jelte/peregrine_mount/100GB.tar-->/testrug/home/[email protected]/iBridges_2023_tests rootResc
[2023-02-15 13:28:15,024] {IrodsConnector.py:1002} INFO - IRODS UPLOADING file /home/jelte/peregrine_mount/100GB.tar to /testrug/home/[email protected]/iBridges_2023_tests/100GB.tar
[2023-02-15 13:28:17,327] {parallel.py:365} INFO - num_threads = 3 ; bytes_per_thread = 33333336746
[2023-02-15 14:36:33,907] {connection.py:121} CRITICAL - The read operation timed out
[2023-02-15 14:36:33,923] {connection.py:122} ERROR - Could not receive server response
[2023-02-15 14:36:33,924] {IrodsConnector.py:1029} INFO - UPLOAD ERROR
Traceback (most recent call last):
File "/home/jelte/python_venvs/ibridges_venv/lib/python3.10/site-packages/irods/message/__init__.py", line 180, in _recv_message_in_len
buf = sock.recv(size_left, socket.MSG_WAITALL)
File "/usr/lib/python3.10/ssl.py", line 1256, in recv
raise ValueError(
ValueError: non-zero flags not allowed in calls to recv() on <class 'ssl.SSLSocket'>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/jelte/python_venvs/ibridges_venv/lib/python3.10/site-packages/irods/connection.py", line 113, in recv
msg = iRODSMessage.recv(self.socket)
File "/home/jelte/python_venvs/ibridges_venv/lib/python3.10/site-packages/irods/message/__init__.py", line 270, in recv
rsp_header_size = _recv_message_in_len(sock, 4)
File "/home/jelte/python_venvs/ibridges_venv/lib/python3.10/site-packages/irods/message/__init__.py", line 182, in _recv_message_in_len
buf = sock.recv(size_left)
File "/usr/lib/python3.10/ssl.py", line 1259, in recv
return self.read(buflen)
File "/usr/lib/python3.10/ssl.py", line 1132, in read
return self._sslobj.read(len)
TimeoutError: The read operation timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/jelte/git_projects/iBridges/utils/IrodsConnector.py", line 1004, in upload_data
self.irods_put(src_path, cmp_path, resc_name)
File "/home/jelte/git_projects/iBridges/utils/IrodsConnector.py", line 851, in irods_put
self.session.data_objects.put(local_path, irods_path, **options)
File "/home/jelte/python_venvs/ibridges_venv/lib/python3.10/site-packages/irods/manager/data_object_manager.py", line 151, in put
if not self.parallel_put( local_path, (obj,o), total_bytes = sizelist[0], num_threads = num_threads,
File "/home/jelte/python_venvs/ibridges_venv/lib/python3.10/site-packages/irods/manager/data_object_manager.py", line 244, in parallel_put
return parallel.io_main( self.sess, data_or_path_, parallel.Oper.PUT | (parallel.Oper.NONBLOCKING if async_ else 0), file_,
File "/home/jelte/python_venvs/ibridges_venv/lib/python3.10/site-packages/irods/parallel.py", line 482, in io_main
retval = _io_multipart_threaded (Operation, (Data, Io), replica_token, resc_hier, session, fname, total_bytes,
File "/home/jelte/python_venvs/ibridges_venv/lib/python3.10/site-packages/irods/parallel.py", line 400, in _io_multipart_threaded
bytecounts = [ f.result() for f in futures ]
File "/home/jelte/python_venvs/ibridges_venv/lib/python3.10/site-packages/irods/parallel.py", line 400, in <listcomp>
bytecounts = [ f.result() for f in futures ]
File "/usr/lib/python3.10/concurrent/futures/_base.py", line 458, in result
return self.__get_result()
File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
['/home/jelte/peregrine_mount/100GB.tar'] 100000010240
100000010240 0
[2023-02-15 13:28:15,016] {IrodsConnector.py:967} INFO - iRODS UPLOAD: /home/jelte/peregrine_mount/100GB.tar-->/testrug/home/[email protected]/iBridges_2023_tests rootResc
[2023-02-15 13:28:15,024] {IrodsConnector.py:1002} INFO - IRODS UPLOADING file /home/jelte/peregrine_mount/100GB.tar to /testrug/home/[email protected]/iBridges_2023_tests/100GB.tar
[2023-02-15 13:28:17,327] {parallel.py:365} INFO - num_threads = 3 ; bytes_per_thread = 33333336746
[2023-02-15 14:36:33,907] {connection.py:121} CRITICAL - The read operation timed out
[2023-02-15 14:36:33,923] {connection.py:122} ERROR - Could not receive server response
[2023-02-15 14:36:33,924] {IrodsConnector.py:1029} INFO - UPLOAD ERROR
Traceback (most recent call last):
File "/home/jelte/python_venvs/ibridges_venv/lib/python3.10/site-packages/irods/message/__init__.py", line 180, in _recv_message_in_len
buf = sock.recv(size_left, socket.MSG_WAITALL)
File "/usr/lib/python3.10/ssl.py", line 1256, in recv
raise ValueError(
ValueError: non-zero flags not allowed in calls to recv() on <class 'ssl.SSLSocket'>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/jelte/python_venvs/ibridges_venv/lib/python3.10/site-packages/irods/connection.py", line 113, in recv
msg = iRODSMessage.recv(self.socket)
File "/home/jelte/python_venvs/ibridges_venv/lib/python3.10/site-packages/irods/message/__init__.py", line 270, in recv
rsp_header_size = _recv_message_in_len(sock, 4)
File "/home/jelte/python_venvs/ibridges_venv/lib/python3.10/site-packages/irods/message/__init__.py", line 182, in _recv_message_in_len
buf = sock.recv(size_left)
File "/usr/lib/python3.10/ssl.py", line 1259, in recv
return self.read(buflen)
File "/usr/lib/python3.10/ssl.py", line 1132, in read
return self._sslobj.read(len)
TimeoutError: The read operation timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/jelte/git_projects/iBridges/utils/IrodsConnector.py", line 1004, in upload_data
self.irods_put(src_path, cmp_path, resc_name)
File "/home/jelte/git_projects/iBridges/utils/IrodsConnector.py", line 851, in irods_put
self.session.data_objects.put(local_path, irods_path, **options)
File "/home/jelte/python_venvs/ibridges_venv/lib/python3.10/site-packages/irods/manager/data_object_manager.py", line 151, in put
if not self.parallel_put( local_path, (obj,o), total_bytes = sizelist[0], num_threads = num_threads,
File "/home/jelte/python_venvs/ibridges_venv/lib/python3.10/site-packages/irods/manager/data_object_manager.py", line 244, in parallel_put
return parallel.io_main( self.sess, data_or_path_, parallel.Oper.PUT | (parallel.Oper.NONBLOCKING if async_ else 0), file_,
File "/home/jelte/python_venvs/ibridges_venv/lib/python3.10/site-packages/irods/parallel.py", line 482, in io_main
retval = _io_multipart_threaded (Operation, (Data, Io), replica_token, resc_hier, session, fname, total_bytes,
File "/home/jelte/python_venvs/ibridges_venv/lib/python3.10/site-packages/irods/parallel.py", line 400, in _io_multipart_threaded
bytecounts = [ f.result() for f in futures ]
File "/home/jelte/python_venvs/ibridges_venv/lib/python3.10/site-packages/irods/parallel.py", line 400, in <listcomp>
bytecounts = [ f.result() for f in futures ]
File "/usr/lib/python3.10/concurrent/futures/_base.py", line 458, in result
return self.__get_result()
File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
raise self._exception
File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/jelte/python_venvs/ibridges_venv/lib/python3.10/site-packages/irods/parallel.py", line 340, in _io_part
return ( _copy_part (file_, objHandle, length, queueObject, thread_debug_id, mgr_) if Operation.isPut()
File "/home/jelte/python_venvs/ibridges_venv/lib/python3.10/site-packages/irods/parallel.py", line 272, in _copy_part
mgr.remove_io( obj_ ) # 1. closes obj if it is not the mgr's initial descriptor
File "/home/jelte/python_venvs/ibridges_venv/lib/python3.10/site-packages/irods/parallel.py", line 320, in remove_io
if is_initial: self.finalize()
File "/home/jelte/python_venvs/ibridges_venv/lib/python3.10/site-packages/irods/parallel.py", line 323, in finalize
self.initial_io.close()
File "/home/jelte/python_venvs/ibridges_venv/lib/python3.10/site-packages/irods/data_object.py", line 177, in close
self.conn.close_file(self.desc, **self.options)
File "/home/jelte/python_venvs/ibridges_venv/lib/python3.10/site-packages/irods/connection.py", line 616, in close_file
self.recv()
File "/home/jelte/python_venvs/ibridges_venv/lib/python3.10/site-packages/irods/connection.py", line 124, in recv
raise NetworkException("Could not receive server response")
irods.exception.NetworkException: Could not receive server response
[2023-02-15 14:36:33,931] {dataTransfer.py:335} INFO - NetworkException('Could not receive server response')
Could not receive server response
Thread stopped
raise self._exception
File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/jelte/python_venvs/ibridges_venv/lib/python3.10/site-packages/irods/parallel.py", line 340, in _io_part
return ( _copy_part (file_, objHandle, length, queueObject, thread_debug_id, mgr_) if Operation.isPut()
File "/home/jelte/python_venvs/ibridges_venv/lib/python3.10/site-packages/irods/parallel.py", line 272, in _copy_part
mgr.remove_io( obj_ ) # 1. closes obj if it is not the mgr's initial descriptor
File "/home/jelte/python_venvs/ibridges_venv/lib/python3.10/site-packages/irods/parallel.py", line 320, in remove_io
if is_initial: self.finalize()
File "/home/jelte/python_venvs/ibridges_venv/lib/python3.10/site-packages/irods/parallel.py", line 323, in finalize
self.initial_io.close()
File "/home/jelte/python_venvs/ibridges_venv/lib/python3.10/site-packages/irods/data_object.py", line 177, in close
self.conn.close_file(self.desc, **self.options)
File "/home/jelte/python_venvs/ibridges_venv/lib/python3.10/site-packages/irods/connection.py", line 616, in close_file
self.recv()
File "/home/jelte/python_venvs/ibridges_venv/lib/python3.10/site-packages/irods/connection.py", line 124, in recv
raise NetworkException("Could not receive server response")
irods.exception.NetworkException: Could not receive server response
[2023-02-15 14:36:33,931] {dataTransfer.py:335} INFO - NetworkException('Could not receive server response')
Could not receive server response
Thread stopped
The file does appear on iRODS without any recognizable error when using icommands:
The checksum also is equivalent to the one calculated for the file on the source.
Trying the same for a smaller, 17 GB archive, results in:
['/home/jelte/Downloads/17GB.tar'] 17000007680
17000007680 0
[2023-02-15 15:03:57,544] {IrodsConnector.py:967} INFO - iRODS UPLOAD: /home/jelte/Downloads/17GB.tar-->/testrug/home/[email protected]/iBridges_2023_tests rootResc
[2023-02-15 15:03:57,550] {IrodsConnector.py:1002} INFO - IRODS UPLOADING file /home/jelte/Downloads/17GB.tar to /testrug/home/[email protected]/iBridges_2023_tests/17GB.tar
[2023-02-15 15:03:58,211] {parallel.py:365} INFO - num_threads = 3 ; bytes_per_thread = 5666669226
[2023-02-15 15:09:23,394] {connection.py:121} CRITICAL - The read operation timed out
[2023-02-15 15:09:23,400] {connection.py:122} ERROR - Could not receive server response
[2023-02-15 15:09:23,400] {IrodsConnector.py:1029} INFO - UPLOAD ERROR
Traceback (most recent call last):
File "/home/jelte/python_venvs/ibridges_venv/lib/python3.10/site-packages/irods/message/__init__.py", line 180, in _recv_message_in_len
buf = sock.recv(size_left, socket.MSG_WAITALL)
File "/usr/lib/python3.10/ssl.py", line 1256, in recv
raise ValueError(
ValueError: non-zero flags not allowed in calls to recv() on <class 'ssl.SSLSocket'>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/jelte/python_venvs/ibridges_venv/lib/python3.10/site-packages/irods/connection.py", line 113, in recv
msg = iRODSMessage.recv(self.socket)
File "/home/jelte/python_venvs/ibridges_venv/lib/python3.10/site-packages/irods/message/__init__.py", line 270, in recv
rsp_header_size = _recv_message_in_len(sock, 4)
File "/home/jelte/python_venvs/ibridges_venv/lib/python3.10/site-packages/irods/message/__init__.py", line 182, in _recv_message_in_len
buf = sock.recv(size_left)
File "/usr/lib/python3.10/ssl.py", line 1259, in recv
return self.read(buflen)
File "/usr/lib/python3.10/ssl.py", line 1132, in read
return self._sslobj.read(len)
TimeoutError: The read operation timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/jelte/git_projects/iBridges/utils/IrodsConnector.py", line 1004, in upload_data
self.irods_put(src_path, cmp_path, resc_name)
File "/home/jelte/git_projects/iBridges/utils/IrodsConnector.py", line 851, in irods_put
self.session.data_objects.put(local_path, irods_path, **options)
File "/home/jelte/python_venvs/ibridges_venv/lib/python3.10/site-packages/irods/manager/data_object_manager.py", line 151, in put
if not self.parallel_put( local_path, (obj,o), total_bytes = sizelist[0], num_threads = num_threads,
File "/home/jelte/python_venvs/ibridges_venv/lib/python3.10/site-packages/irods/manager/data_object_manager.py", line 244, in parallel_put
return parallel.io_main( self.sess, data_or_path_, parallel.Oper.PUT | (parallel.Oper.NONBLOCKING if async_ else 0), file_,
File "/home/jelte/python_venvs/ibridges_venv/lib/python3.10/site-packages/irods/parallel.py", line 482, in io_main
retval = _io_multipart_threaded (Operation, (Data, Io), replica_token, resc_hier, session, fname, total_bytes,
File "/home/jelte/python_venvs/ibridges_venv/lib/python3.10/site-packages/irods/parallel.py", line 400, in _io_multipart_threaded
bytecounts = [ f.result() for f in futures ]
File "/home/jelte/python_venvs/ibridges_venv/lib/python3.10/site-packages/irods/parallel.py", line 400, in <listcomp>
bytecounts = [ f.result() for f in futures ]
File "/usr/lib/python3.10/concurrent/futures/_base.py", line 458, in result
return self.__get_result()
File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
raise self._exception
File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/jelte/python_venvs/ibridges_venv/lib/python3.10/site-packages/irods/parallel.py", line 340, in _io_part
return ( _copy_part (file_, objHandle, length, queueObject, thread_debug_id, mgr_) if Operation.isPut()
File "/home/jelte/python_venvs/ibridges_venv/lib/python3.10/site-packages/irods/parallel.py", line 272, in _copy_part
mgr.remove_io( obj_ ) # 1. closes obj if it is not the mgr's initial descriptor
File "/home/jelte/python_venvs/ibridges_venv/lib/python3.10/site-packages/irods/parallel.py", line 320, in remove_io
if is_initial: self.finalize()
File "/home/jelte/python_venvs/ibridges_venv/lib/python3.10/site-packages/irods/parallel.py", line 323, in finalize
self.initial_io.close()
File "/home/jelte/python_venvs/ibridges_venv/lib/python3.10/site-packages/irods/data_object.py", line 177, in close
self.conn.close_file(self.desc, **self.options)
File "/home/jelte/python_venvs/ibridges_venv/lib/python3.10/site-packages/irods/connection.py", line 616, in close_file
self.recv()
File "/home/jelte/python_venvs/ibridges_venv/lib/python3.10/site-packages/irods/connection.py", line 124, in recv
raise NetworkException("Could not receive server response")
irods.exception.NetworkException: Could not receive server response
[2023-02-15 15:09:23,411] {dataTransfer.py:335} INFO - NetworkException('Could not receive server response')
Could not receive server response
Thread stopped
Again, the file seems to arrive successfully on iRODS, and the checksums between destination and source are identical.
I also tried to use the Upload function in the normal file browser tab for comparison, bust just for the 17GB archive. The client first seems to freeze during the transfer, but then displays the following message:
[2023-02-15 15:25:34,119] {IrodsConnector.py:967} INFO - iRODS UPLOAD: /home/jelte/Downloads/17GB.tar-->/testrug/home/[email protected]
[2023-02-15 15:25:34,135] {IrodsConnector.py:1002} INFO - IRODS UPLOADING file /home/jelte/Downloads/17GB.tar to /testrug/home/[email protected]/17GB.tar
[2023-02-15 15:25:35,461] {parallel.py:365} INFO - num_threads = 3 ; bytes_per_thread = 5666669226
[2023-02-15 15:30:07,093] {connection.py:121} CRITICAL - The read operation timed out
[2023-02-15 15:30:07,099] {connection.py:122} ERROR - Could not receive server response
[2023-02-15 15:30:07,099] {IrodsConnector.py:1029} INFO - UPLOAD ERROR
Traceback (most recent call last):
File "/home/jelte/python_venvs/ibridges_venv/lib/python3.10/site-packages/irods/message/__init__.py", line 180, in _recv_message_in_len
buf = sock.recv(size_left, socket.MSG_WAITALL)
File "/usr/lib/python3.10/ssl.py", line 1256, in recv
raise ValueError(
ValueError: non-zero flags not allowed in calls to recv() on <class 'ssl.SSLSocket'>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/jelte/python_venvs/ibridges_venv/lib/python3.10/site-packages/irods/connection.py", line 113, in recv
msg = iRODSMessage.recv(self.socket)
File "/home/jelte/python_venvs/ibridges_venv/lib/python3.10/site-packages/irods/message/__init__.py", line 270, in recv
rsp_header_size = _recv_message_in_len(sock, 4)
File "/home/jelte/python_venvs/ibridges_venv/lib/python3.10/site-packages/irods/message/__init__.py", line 182, in _recv_message_in_len
buf = sock.recv(size_left)
File "/usr/lib/python3.10/ssl.py", line 1259, in recv
return self.read(buflen)
File "/usr/lib/python3.10/ssl.py", line 1132, in read
return self._sslobj.read(len)
TimeoutError: The read operation timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/jelte/git_projects/iBridges/utils/IrodsConnector.py", line 1004, in upload_data
self.irods_put(src_path, cmp_path, resc_name)
File "/home/jelte/git_projects/iBridges/utils/IrodsConnector.py", line 851, in irods_put
self.session.data_objects.put(local_path, irods_path, **options)
File "/home/jelte/python_venvs/ibridges_venv/lib/python3.10/site-packages/irods/manager/data_object_manager.py", line 151, in put
if not self.parallel_put( local_path, (obj,o), total_bytes = sizelist[0], num_threads = num_threads,
File "/home/jelte/python_venvs/ibridges_venv/lib/python3.10/site-packages/irods/manager/data_object_manager.py", line 244, in parallel_put
return parallel.io_main( self.sess, data_or_path_, parallel.Oper.PUT | (parallel.Oper.NONBLOCKING if async_ else 0), file_,
File "/home/jelte/python_venvs/ibridges_venv/lib/python3.10/site-packages/irods/parallel.py", line 482, in io_main
retval = _io_multipart_threaded (Operation, (Data, Io), replica_token, resc_hier, session, fname, total_bytes,
File "/home/jelte/python_venvs/ibridges_venv/lib/python3.10/site-packages/irods/parallel.py", line 400, in _io_multipart_threaded
bytecounts = [ f.result() for f in futures ]
File "/home/jelte/python_venvs/ibridges_venv/lib/python3.10/site-packages/irods/parallel.py", line 400, in <listcomp>
bytecounts = [ f.result() for f in futures ]
File "/usr/lib/python3.10/concurrent/futures/_base.py", line 458, in result
return self.__get_result()
File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
raise self._exception
File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/jelte/python_venvs/ibridges_venv/lib/python3.10/site-packages/irods/parallel.py", line 340, in _io_part
return ( _copy_part (file_, objHandle, length, queueObject, thread_debug_id, mgr_) if Operation.isPut()
File "/home/jelte/python_venvs/ibridges_venv/lib/python3.10/site-packages/irods/parallel.py", line 272, in _copy_part
mgr.remove_io( obj_ ) # 1. closes obj if it is not themgr's initial descriptor
File "/home/jelte/python_venvs/ibridges_venv/lib/python3.10/site-packages/irods/parallel.py", line 320, in remove_io
if is_initial: self.finalize()
File "/home/jelte/python_venvs/ibridges_venv/lib/python3.10/site-packages/irods/parallel.py", line 323, in finalize
self.initial_io.close()
File "/home/jelte/python_venvs/ibridges_venv/lib/python3.10/site-packages/irods/data_object.py", line 177, in close
self.conn.close_file(self.desc, **self.options)
File "/home/jelte/python_venvs/ibridges_venv/lib/python3.10/site-packages/irods/connection.py", line 616, in close_file
self.recv()
File "/home/jelte/python_venvs/ibridges_venv/lib/python3.10/site-packages/irods/connection.py", line 124, in recv
raise NetworkException("Could not receive server response")
irods.exception.NetworkException: Could not receive server response
Exception ignored in: <function Connection.__del__ at 0x7f06d3254280>
Traceback (most recent call last):
File "/home/jelte/python_venvs/ibridges_venv/lib/python3.10/site-packages/irods/connection.py", line 89, in __del__
self.disconnect()
File "/home/jelte/python_venvs/ibridges_venv/lib/python3.10/site-packages/irods/connection.py", line 299, in disconnect
self.socket = self.socket.unwrap()
File "/usr/lib/python3.10/ssl.py", line 1318, in unwrap
s = self._sslobj.shutdown()
ssl.SSLError: [SSL: APPLICATION_DATA_AFTER_CLOSE_NOTIFY] application data after close notify (_ssl.c:2672)
As above, the files appear on iRODS and checksum are identical.
The text was updated successfully, but these errors were encountered:
Using the current develop branch (commit f022d06), I saw some issues when trying to transfer larger files. While file transfers for smaller files worked fine, I saw issues when trying upload a 17GB and a 100GB tar archive.
First some general info:
Python version used:
Python 3.10.6
All other required Python packages are in the version specified in the
requirements.txt
The issue in more detail:
Data Transfers
tabAfter a longer time, the client crashes and the following message is displayed:
The file does appear on iRODS without any recognizable error when using
icommands
:The checksum also is equivalent to the one calculated for the file on the source.
Again, the file seems to arrive successfully on iRODS, and the checksums between destination and source are identical.
As above, the files appear on iRODS and checksum are identical.
The text was updated successfully, but these errors were encountered: