Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cap maximum shard size at the size of an integer #5141

Merged
merged 2 commits into from
Jul 29, 2021
Merged

Cap maximum shard size at the size of an integer #5141

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
8bf63b3
Cap maximum shard size at the size of an integer
mrocklin Jul 29, 2021
f394060
Update distributed/comm/tcp.py
mrocklin Jul 29, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions distributed/comm/tcp.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -145,6 +145,8 @@ class TCP(Comm):
An established communication based on an underlying Tornado IOStream.
"""

max_shard_size = dask.utils.parse_bytes(dask.config.get("distributed.comm.shard"))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems to be 64 MiB. Do we want to limit it to that? Would it be worthwhile to have a different value for TCP specifically?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR doesn't change the behavior for TCP (this value is what is used by default for everything in to_frames). All it does is make sure that TLS has a possibly different value that is capped by the max int size, which I think was the intent of the other PR.

Copy link
Member

@jakirkham jakirkham Jul 29, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it. So this was 64 MiB before?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup.


def __init__(self, stream, local_addr, peer_addr, deserialize=True):
self._closed = False
Comm.__init__(self)
Expand Down Expand Up @@ -248,6 +250,7 @@ async def write(self, msg, serializers=None, on_error="message"):
"recipient": self.remote_info,
**self.handshake_options,
},
frame_split_size=self.max_shard_size,
)
frames_nbytes = [nbytes(f) for f in frames]
frames_nbytes_total = sum(frames_nbytes)
Expand Down Expand Up @@ -335,6 +338,9 @@ class TLS(TCP):
A TLS-specific version of TCP.
"""

# Workaround for OpenSSL 1.0.2 (can drop with OpenSSL 1.1.1)
max_shard_size = min(C_INT_MAX, TCP.max_shard_size)
mrocklin marked this conversation as resolved.
Show resolved Hide resolved

def _read_extra(self):
TCP._read_extra(self)
sock = self.stream.socket
Expand Down