Skip to content

Commit

Permalink
Fix calculation of MemoryShardsBuffer.bytes_read
Browse files Browse the repository at this point in the history
  • Loading branch information
crusaderky committed Oct 19, 2023
1 parent cbc3a33 commit 5bef354
Showing 1 changed file with 5 additions and 8 deletions.
13 changes: 5 additions & 8 deletions distributed/shuffle/_memory.py
Original file line number Diff line number Diff line change
Expand Up @@ -34,16 +34,13 @@ def read(self, id: str) -> Any:
raise RuntimeError("Tried to read from file before done.")

with self.time("read"):
shards = self._shards.pop(id) # Raises KeyError
self.bytes_read += sum(map(sizeof, shards))

Check warning on line 38 in distributed/shuffle/_memory.py

View check run for this annotation

Codecov / codecov/patch

distributed/shuffle/_memory.py#L37-L38

Added lines #L37 - L38 were not covered by tests
# Don't keep the serialized and the deserialized shards
# in memory at the same time
data = []
size = 0
shards = self._shards[id]
while shards:
shard = shards.pop()
data.append(self._deserialize(shard))
size += sizeof(shards)

if data:
self.bytes_read += size
return data
else:
raise KeyError(id)
return data

Check warning on line 46 in distributed/shuffle/_memory.py

View check run for this annotation

Codecov / codecov/patch

distributed/shuffle/_memory.py#L46

Added line #L46 was not covered by tests

0 comments on commit 5bef354

Please sign in to comment.