Flyte v0.5.0 release #374
Assignees
Milestone
Comments
Merged
|
This is now live |
eapolinario
pushed a commit
to eapolinario/flyte
that referenced
this issue
Dec 6, 2022
* Scale out with propeller manager and workflow sharding (flyteorg#351) * added 'manager' command Signed-off-by: Daniel Rammer <[email protected]> * using go routine and timer for manager loop Signed-off-by: Daniel Rammer <[email protected]> * moved manager loop out of cmd and into pkg directory Signed-off-by: Daniel Rammer <[email protected]> * detecting missing replicas Signed-off-by: Daniel Rammer <[email protected]> * moved extracting replica from pod name to new function Signed-off-by: Daniel Rammer <[email protected]> * creating managed flytepropeller pods Signed-off-by: Daniel Rammer <[email protected]> * refactored configuration Signed-off-by: Daniel Rammer <[email protected]> * removed regex parsing for replica - checking for existance with fully qualified pod name Signed-off-by: Daniel Rammer <[email protected]> * mocked out shard strategy abstraction Signed-off-by: Daniel Rammer <[email protected]> * adding arguments to podspec for ConsistentHashingShardStrategy Signed-off-by: Daniel Rammer <[email protected]> * updated import naming Signed-off-by: Daniel Rammer <[email protected]> * moved manager to a top-level package Signed-off-by: Daniel Rammer <[email protected]> * added shard strategy to manager configuration Signed-off-by: Daniel Rammer <[email protected]> * setting shard key label selector on managed propeller instances Signed-off-by: Daniel Rammer <[email protected]> * fixed random lint issues Signed-off-by: Daniel Rammer <[email protected]> * split pod name generate to separate function to ease future auto-scaler implementation Signed-off-by: Daniel Rammer <[email protected]> * cleaned up pod label selector Signed-off-by: Daniel Rammer <[email protected]> * delete pods on shutdown Signed-off-by: Daniel Rammer <[email protected]> * added prometheus metric reporting Signed-off-by: Daniel Rammer <[email protected]> * updated manager run loop to use k8s wait.UntilWithContext Signed-off-by: Daniel Rammer <[email protected]> * moved getKubeConfig into a shared package Signed-off-by: Daniel Rammer <[email protected]> * assigning shard and namespace labels on FlyteWorkflow Signed-off-by: Daniel Rammer <[email protected]> * implement NamespaceShardStrategy Signed-off-by: Daniel Rammer <[email protected]> * implemented NamespaceShardStrategy Signed-off-by: Daniel Rammer <[email protected]> * fixed shard label Signed-off-by: Daniel Rammer <[email protected]> * added comments Signed-off-by: Daniel Rammer <[email protected]> * checking for existing pods on startup Signed-off-by: Daniel Rammer <[email protected]> * handling delete of non-existent pod Signed-off-by: Daniel Rammer <[email protected]> * changes ConsistentHashing name to Random - because that's what it really is Signed-off-by: Daniel Rammer <[email protected]> * implemented EnableUncoveredReplica configuration option Signed-off-by: Daniel Rammer <[email protected]> * added leader election to manager using existing propeller config Signed-off-by: Daniel Rammer <[email protected]> * fixed disable leader election in managed propeller pods Signed-off-by: Daniel Rammer <[email protected]> * removed listPods function Signed-off-by: Daniel Rammer <[email protected]> * added leader election to mitigate concurrent modification issues Signed-off-by: Daniel Rammer <[email protected]> * enabled pprof to profile resource metrics Signed-off-by: Daniel Rammer <[email protected]> * added 'manager' target to Makefile to start manager in development mode (similar to existing server) Signed-off-by: Daniel Rammer <[email protected]> * added shard strategy test for computing key ranges Signed-off-by: Daniel Rammer <[email protected]> * fixed key range computation Signed-off-by: Daniel Rammer <[email protected]> * implemented project and domain shard types Signed-off-by: Daniel Rammer <[email protected]> * returning error on out of range podIndex during UpdatePodSpec call on shard strategy Signed-off-by: Daniel Rammer <[email protected]> * fixed random lint issues Signed-off-by: Daniel Rammer <[email protected]> * added manager tests Signed-off-by: Daniel Rammer <[email protected]> * fixed lint issues Signed-off-by: Daniel Rammer <[email protected]> * added doc comments on exported types and functions Signed-off-by: Daniel Rammer <[email protected]> * exporting ComputeKeyRange function and changed adding addLabelSelector function name to addLabelSelectorIfExists to better reflect functionality Signed-off-by: Daniel Rammer <[email protected]> * adding pod template resource version and shard config hash annotations to fuel automatic pod management on updates Signed-off-by: Daniel Rammer <[email protected]> * removed pod deletion on manager shutdown Signed-off-by: Daniel Rammer <[email protected]> * cleaned up unit tests and lint Signed-off-by: Daniel Rammer <[email protected]> * updated getContainer function to retrive flytepropeller container from pod spec using container name instead of command Signed-off-by: Daniel Rammer <[email protected]> * removed addLabelSelectorIfExists function call Signed-off-by: Daniel Rammer <[email protected]> * changed bytes.Buffer from a var to declaring with new Signed-off-by: Daniel Rammer <[email protected]> * created a new shardstrategy package Signed-off-by: Daniel Rammer <[email protected]> * generating mocks for ShardStrategy to decouple manager package tests from shardstrategy package tests Signed-off-by: Daniel Rammer <[email protected]> * fixed lint issues Signed-off-by: Daniel Rammer <[email protected]> * changed shard configuration defintions and added support for wildcard id in EnvironmentShardStrategy Signed-off-by: Daniel Rammer <[email protected]> * updated documentation Signed-off-by: Daniel Rammer <[email protected]> * fixed lint issues Signed-off-by: Daniel Rammer <[email protected]> * setting managed pod owner references Signed-off-by: Daniel Rammer <[email protected]> * updated documentation Signed-off-by: Daniel Rammer <[email protected]> * fixed a few nits Signed-off-by: Daniel Rammer <[email protected]> * delete pods with failed state Signed-off-by: Daniel Rammer <[email protected]> * changed ShardType type to int instead of string Signed-off-by: Daniel Rammer <[email protected]> * removed default values in manager config Signed-off-by: Daniel Rammer <[email protected]> * updated config_flags with pflags generation Signed-off-by: Daniel Rammer <[email protected]> Signed-off-by: Haytham Abuelfutuh <[email protected]> * Create codeql-analysis.yml Signed-off-by: Haytham Abuelfutuh <[email protected]> * Handle code quality issue Signed-off-by: Haytham Abuelfutuh <[email protected]> * check boundaries Signed-off-by: Haytham Abuelfutuh <[email protected]> * 0 is ok Signed-off-by: Haytham Abuelfutuh <[email protected]> * Use ParseUint instead Signed-off-by: Haytham Abuelfutuh <[email protected]> * bump for DCO Signed-off-by: Haytham Abuelfutuh <[email protected]> Co-authored-by: Dan Rammer <[email protected]>
eapolinario
pushed a commit
to eapolinario/flyte
that referenced
this issue
Dec 20, 2022
…fo (flyteorg#374) * fix: service account is not properly propagated from the execution info Signed-off-by: eugenejahn <[email protected]>
eapolinario
pushed a commit
to eapolinario/flyte
that referenced
this issue
Dec 20, 2022
## [0.51.2](http://github.com/lyft/flyteconsole/compare/v0.51.1...v0.51.2) (2022-04-13) ### Bug Fixes * service account is not properly propagated from the execution info ([flyteorg#374](http://github.com/lyft/flyteconsole/issues/374)) ([2e250bb](http://github.com/lyft/flyteconsole/commit/2e250bb53159069bc34b8578ac53aa64334837ce))
eapolinario
pushed a commit
to eapolinario/flyte
that referenced
this issue
Aug 9, 2023
* Scale out with propeller manager and workflow sharding (flyteorg#351) * added 'manager' command Signed-off-by: Daniel Rammer <[email protected]> * using go routine and timer for manager loop Signed-off-by: Daniel Rammer <[email protected]> * moved manager loop out of cmd and into pkg directory Signed-off-by: Daniel Rammer <[email protected]> * detecting missing replicas Signed-off-by: Daniel Rammer <[email protected]> * moved extracting replica from pod name to new function Signed-off-by: Daniel Rammer <[email protected]> * creating managed flytepropeller pods Signed-off-by: Daniel Rammer <[email protected]> * refactored configuration Signed-off-by: Daniel Rammer <[email protected]> * removed regex parsing for replica - checking for existance with fully qualified pod name Signed-off-by: Daniel Rammer <[email protected]> * mocked out shard strategy abstraction Signed-off-by: Daniel Rammer <[email protected]> * adding arguments to podspec for ConsistentHashingShardStrategy Signed-off-by: Daniel Rammer <[email protected]> * updated import naming Signed-off-by: Daniel Rammer <[email protected]> * moved manager to a top-level package Signed-off-by: Daniel Rammer <[email protected]> * added shard strategy to manager configuration Signed-off-by: Daniel Rammer <[email protected]> * setting shard key label selector on managed propeller instances Signed-off-by: Daniel Rammer <[email protected]> * fixed random lint issues Signed-off-by: Daniel Rammer <[email protected]> * split pod name generate to separate function to ease future auto-scaler implementation Signed-off-by: Daniel Rammer <[email protected]> * cleaned up pod label selector Signed-off-by: Daniel Rammer <[email protected]> * delete pods on shutdown Signed-off-by: Daniel Rammer <[email protected]> * added prometheus metric reporting Signed-off-by: Daniel Rammer <[email protected]> * updated manager run loop to use k8s wait.UntilWithContext Signed-off-by: Daniel Rammer <[email protected]> * moved getKubeConfig into a shared package Signed-off-by: Daniel Rammer <[email protected]> * assigning shard and namespace labels on FlyteWorkflow Signed-off-by: Daniel Rammer <[email protected]> * implement NamespaceShardStrategy Signed-off-by: Daniel Rammer <[email protected]> * implemented NamespaceShardStrategy Signed-off-by: Daniel Rammer <[email protected]> * fixed shard label Signed-off-by: Daniel Rammer <[email protected]> * added comments Signed-off-by: Daniel Rammer <[email protected]> * checking for existing pods on startup Signed-off-by: Daniel Rammer <[email protected]> * handling delete of non-existent pod Signed-off-by: Daniel Rammer <[email protected]> * changes ConsistentHashing name to Random - because that's what it really is Signed-off-by: Daniel Rammer <[email protected]> * implemented EnableUncoveredReplica configuration option Signed-off-by: Daniel Rammer <[email protected]> * added leader election to manager using existing propeller config Signed-off-by: Daniel Rammer <[email protected]> * fixed disable leader election in managed propeller pods Signed-off-by: Daniel Rammer <[email protected]> * removed listPods function Signed-off-by: Daniel Rammer <[email protected]> * added leader election to mitigate concurrent modification issues Signed-off-by: Daniel Rammer <[email protected]> * enabled pprof to profile resource metrics Signed-off-by: Daniel Rammer <[email protected]> * added 'manager' target to Makefile to start manager in development mode (similar to existing server) Signed-off-by: Daniel Rammer <[email protected]> * added shard strategy test for computing key ranges Signed-off-by: Daniel Rammer <[email protected]> * fixed key range computation Signed-off-by: Daniel Rammer <[email protected]> * implemented project and domain shard types Signed-off-by: Daniel Rammer <[email protected]> * returning error on out of range podIndex during UpdatePodSpec call on shard strategy Signed-off-by: Daniel Rammer <[email protected]> * fixed random lint issues Signed-off-by: Daniel Rammer <[email protected]> * added manager tests Signed-off-by: Daniel Rammer <[email protected]> * fixed lint issues Signed-off-by: Daniel Rammer <[email protected]> * added doc comments on exported types and functions Signed-off-by: Daniel Rammer <[email protected]> * exporting ComputeKeyRange function and changed adding addLabelSelector function name to addLabelSelectorIfExists to better reflect functionality Signed-off-by: Daniel Rammer <[email protected]> * adding pod template resource version and shard config hash annotations to fuel automatic pod management on updates Signed-off-by: Daniel Rammer <[email protected]> * removed pod deletion on manager shutdown Signed-off-by: Daniel Rammer <[email protected]> * cleaned up unit tests and lint Signed-off-by: Daniel Rammer <[email protected]> * updated getContainer function to retrive flytepropeller container from pod spec using container name instead of command Signed-off-by: Daniel Rammer <[email protected]> * removed addLabelSelectorIfExists function call Signed-off-by: Daniel Rammer <[email protected]> * changed bytes.Buffer from a var to declaring with new Signed-off-by: Daniel Rammer <[email protected]> * created a new shardstrategy package Signed-off-by: Daniel Rammer <[email protected]> * generating mocks for ShardStrategy to decouple manager package tests from shardstrategy package tests Signed-off-by: Daniel Rammer <[email protected]> * fixed lint issues Signed-off-by: Daniel Rammer <[email protected]> * changed shard configuration defintions and added support for wildcard id in EnvironmentShardStrategy Signed-off-by: Daniel Rammer <[email protected]> * updated documentation Signed-off-by: Daniel Rammer <[email protected]> * fixed lint issues Signed-off-by: Daniel Rammer <[email protected]> * setting managed pod owner references Signed-off-by: Daniel Rammer <[email protected]> * updated documentation Signed-off-by: Daniel Rammer <[email protected]> * fixed a few nits Signed-off-by: Daniel Rammer <[email protected]> * delete pods with failed state Signed-off-by: Daniel Rammer <[email protected]> * changed ShardType type to int instead of string Signed-off-by: Daniel Rammer <[email protected]> * removed default values in manager config Signed-off-by: Daniel Rammer <[email protected]> * updated config_flags with pflags generation Signed-off-by: Daniel Rammer <[email protected]> Signed-off-by: Haytham Abuelfutuh <[email protected]> * Create codeql-analysis.yml Signed-off-by: Haytham Abuelfutuh <[email protected]> * Handle code quality issue Signed-off-by: Haytham Abuelfutuh <[email protected]> * check boundaries Signed-off-by: Haytham Abuelfutuh <[email protected]> * 0 is ok Signed-off-by: Haytham Abuelfutuh <[email protected]> * Use ParseUint instead Signed-off-by: Haytham Abuelfutuh <[email protected]> * bump for DCO Signed-off-by: Haytham Abuelfutuh <[email protected]> Co-authored-by: Dan Rammer <[email protected]>
eapolinario
pushed a commit
that referenced
this issue
Sep 26, 2023
* Add failing test Signed-off-by: Bernhard Stadlbauer <[email protected]> * WIP Signed-off-by: Bernhard Stadlbauer <[email protected]> * Improve test Signed-off-by: Bernhard Stadlbauer <[email protected]> * Refactor to use `ToK8sPodSpec` Signed-off-by: Bernhard Stadlbauer <[email protected]> * Fix linting Signed-off-by: Bernhard Stadlbauer <[email protected]> * Use `Always` restart policy for workers Signed-off-by: Bernhard Stadlbauer <[email protected]> * Add test which checks whether labels are propagated Signed-off-by: Bernhard Stadlbauer <[email protected]> * Replace `removeInterruptibleConfig` with `TaskExectuionMetadata` wrapper Signed-off-by: Bernhard Stadlbauer <[email protected]> --------- Signed-off-by: Bernhard Stadlbauer <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Propeller: v0.2.64
Admin: v0.2.11
Flytekit: v0.10.0
Console: v0.5.3
IDL: 0.17.34
The text was updated successfully, but these errors were encountered: