[Core feature] Add Container Configuration to Default PodTemplate

[hamersaw]

Motivation: Why do you think this is important?

Currently the default PodTemplate in FlytePropeller allows users to set a base PodSpec configuration for all k8s Pods started by Flyte. This is very flexible in supporting all Pod configuration options without requiring Flyte updates. However, this configuration does not extend into specific Containers. In the current approach, Flyte overrides the default PodTemplate containers during initialization.

Goal: What should the final outcome look like, ideally?

Users should be able to add configuration to the default PodTemplate containers that gets applied to containers in Flyte Pods. This improves the configurability of Flyte. One approach is to use the container names in the PodTemplate to map to Flyte Pod container names. For example a container in the PodTemplate named "primary_container_name" would serve as the base to all primary containers in k8s container and pod plugins.

Describe alternatives you've considered

Leave container configuration unsupported or continue adding one-off configuration options to the k8s plugin to support container specific configuration options. Both of these options is undesired.

Propose: Link/Inline OR Additional context

No response

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes

[hamersaw]

cc @niliayu