Improve documentation of message-bytes-limit
#7077
Conversation
| #: :meth:`BaseWorker.gather_dep`. Multiple small tasks that can be fetched from the | ||
| #: same worker will be clustered in a single instruction as long as their combined | ||
| #: size doesn't exceed this value. | ||
| #: Number of bytes to gather from the same worker in a single call to |
There was a problem hiding this comment.
Driveby: Highlight the fine print here as well.
Unit Test ResultsSee test report for an extended history of previous test failures. This is useful for diagnosing flaky tests. 15 files ±0 15 suites ±0 6h 29m 10s ⏱️ + 5m 54s For more details on these failures, see this check. Results for commit a5e5f57. ± Comparison against base commit 8c4133c. |
| Tasks are gathered in batches, and if the first task in a batch is larger than this value, | ||
| the task will still be gathered to ensure progress. Hence, this limit is not absolute. | ||
| Note that this limit applies to a single ``GatherDep`` and a worker may gather data from | ||
| multiple workers in parallel. |
There was a problem hiding this comment.
The GatherDep class is an implementation detail and shouldn't appear in what is effectively functional documentation. Could you rephrase? e.g. replace ``GatherDep``` with "data transfer between workers"?
There was a problem hiding this comment.
I've replaced GatherDep with gathering operation. This maps to the nomenclature we use in our documentation for the worker state machine. A data transfer may be falsely interpreted as a transfer on the comm level, i.e., the transfer of an individual frame, which in turn has its own configuration variable. This is also why I'm not a huge fan of the message- prefix of this configuration variable (as noted in the PR description).
Addresses
message-bytes-limitin config #7074 (comment)Note
It just occurred to me that it might be even better to rename
message-bytes-limittogather-bytes-limitandtransfer_message_bytes_limittotransfer_gather_bytes_limitto highlight their intention, but I don't want to keep changing names all the time.pre-commit run --all-filescc @wence-, @crusaderky