You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hello, is it possible to explicitly assign elements to workers? I have a process that involves a time-consuming function being run on (currently) 57 elements. Within that some elements take a long time and some are quick, and I can estimate in advance which ones will be slow or fast.
I'd like to determine my own run order that balances the jobs across all the workers and controls the order within each worker to avoid all my workers running a "large" element at the same time (which can cause out of memory issues).
I have tried simply ordering the elements myself and then using with static load balancing to get this to work approximately, but unless the number of elements is an exact multiple of the number of workers it doesn't always work out because I can't work out how the extra elements are assigned to the workers. E.g. with 57 elements and 5 workers I have 11 elements per worker with 2 left over, and I can't work out which workers will get these 2 extras so I can't factor that in to my ordering algorithm.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
Hello, is it possible to explicitly assign elements to workers? I have a process that involves a time-consuming function being run on (currently) 57 elements. Within that some elements take a long time and some are quick, and I can estimate in advance which ones will be slow or fast.
I'd like to determine my own run order that balances the jobs across all the workers and controls the order within each worker to avoid all my workers running a "large" element at the same time (which can cause out of memory issues).
I have tried simply ordering the elements myself and then using with static load balancing to get this to work approximately, but unless the number of elements is an exact multiple of the number of workers it doesn't always work out because I can't work out how the extra elements are assigned to the workers. E.g. with 57 elements and 5 workers I have 11 elements per worker with 2 left over, and I can't work out which workers will get these 2 extras so I can't factor that in to my ordering algorithm.
Beta Was this translation helpful? Give feedback.
All reactions