[flytekit] Revisit retry behavior

[wild-endeavor]

Motivation: Why do you think this is important?
Users do not find the current retry behavior intuitive. Retry behavior in the native API was recently changed to reflect the old API, namely

• Exceptions raised when user code is running are considered non-retriable unless it's a FlyteRecoverableException.
• System exceptions are always retriable.

Goal: What should the final outcome look like, ideally?
Suggestions from the thread include

• Have user errors count towards user-specified task retry count, and system errors count towards the globally set system retry count https://github.com/flyteorg/flytepropeller/blob/65d63de16b0ea5c43f216b2cae3f20b406bd7de2/pkg/controller/config/config.go#L188
• Using a retry library instead of the current implementation.
• Invert so that every user exception is retriable unless specified to be non-recoverable.
• Allow users to specify a retry-strategy

[kumare3]

I like this issue atleast the essence of it, keeping it open