Add rate limiter configs for Presto plugin #75
Conversation
lu4nm3
changed the title
Co-Authored-By: Yee Hing Tong <[email protected]>
| @@ -52,6 +57,10 @@ var ( | |||
| Workers: 15, | |||
| LruCacheSize: 10000, | |||
| }, | |||
| RateLimiterConfig: RateLimiterConfig{ | |||
| Rate: 15, | |||
There was a problem hiding this comment.
What will this be used for? Will it gate all calls to Mozart? Or just Presto? Will it be for only creation calls? Or both creation calls and all the status checking calls in the auto-refresh cache as well? Can you add some comments about how best to calculate a good number for this config?
There was a problem hiding this comment.
This will be used for all calls going to Mozart only for Presto. Sure I can add some comments.
There was a problem hiding this comment.
In terms of the values themselves, I sort of took them from the awsbatch rate limiter. I'm sure we can adjust them later based on how the Presto cluster behaves
|
Can you create and link issue as well? Or use an existing issue? |
|
Made changes from feedback and linked issue |
* Add rate limiter configs * add extern storage location * lint * undo * undo2 * update field description * Update go/tasks/plugins/presto/config/config.go Co-Authored-By: Yee Hing Tong <[email protected]> * add comments in field description for rate limiter Co-authored-by: Yee Hing Tong <[email protected]>
TL;DR
GetRawOutputPrefixType
Are all requirements met?
Issue
flyteorg/flyte#240