Spark: Update Hadoop/AWS Sdk version + set all to Spark3

[akhurana001]

TL;DR

• Updates Hadoop/AWS version to support IAM role fetch via https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-minimum-sdk.html
• Default flytekit[all] to Spark 3.x (Default flytekit[all] to Pyspark 3.x flyte#483)

Type

• Bug Fix
• Feature
• Plugin

Are all requirements met?

• Code completed
• Smoke tested
• Unit tests added
• Code documentation added
• Any pending items have an associated Issue

Complete description

How did you fix the bug, make the feature etc. Link to any design docs etc

Tracking Issue

https://github.com/lyft/flyte/issues/

Follow-up issue

NA
OR
https://github.com/lyft/flyte/issues/

[wild-endeavor]

Can you change this to be a PR into the annotations branch?

Also, on that branch, this information is currently in two places - the lazy loader file, and also the main setup.py file. It'll be merged back shortly but could you just change it in both places for now?

[akhurana001]

Can you change this to be a PR into the annotations branch?

Also, on that branch, this information is currently in two places - the lazy loader file, and also the main setup.py file. It'll be merged back shortly but could you just change it in both places for now?

Created #334 for annotations branch

[akhurana001]

closing in favor of https://github.com/lyft/flytekit/pull/334/files