YandexCloud provider: Support new Yandex SDK features for DataProc

[Piatachock]

Support following parameters in DataprocCreateClusterOperator:

• properties
• initialization_actions
• enable_ui_proxy
• security_group_pds

Support creation of DataProc clusters with no datanodes and S3 instead of HDFS

Add example for direct usage of yandexcloud SDK connection in custom operators

---

Functionality is tested on local AirFlow installation.

[Piatachock]

@potiuk hey Jarek, I'm not sure what to do with CI, guide me please.
I've got a dependency version update in this PR, so I updated provider.yaml file. Now CI fails on check on generated/provider_dependencies.yaml file: https://github.com/apache/airflow/runs/7428106267?check_suite_focus=true

Based on README in generated/ dir, I'm not sure if I am supposed to modify this file by hand. What would you suggest to do?

[potiuk]

You should simply run pre-commit and commit what has been generated.

[potiuk]

It's actually explained at the bottom of the error you see in static checks and here are the docs: https://github.com/apache/airflow/blob/main/STATIC_CODE_CHECKS.rst#pre-commit-hooks

[potiuk]

And feel free to add in the README that you should commit the changes generated by pre-commit (and also you could add link to https://github.com/apache/airflow/blob/main/STATIC_CODE_CHECKS.rst#pre-commit-hooks ). Ideally you are the best person to phrase it correctly as you misunderstood what I was trying to tell there - so if you can improve the README description that woudl be fantastic.

[Piatachock]

That makes sense, thanks! I guess I was not sure if pre-commit should happen on my side or on CI side.
I'll see if there is other way to phrase it in README

[potiuk]

Rebased to account for Werkzeug failures, but likely there are tests to address anyway

[Piatachock]

Not sure if fails are related to changes of this PR;

Unknown warnings generated:
[1393](https://github.com/apache/airflow/runs/7520766090?check_suite_focus=true#step:12:1400)
  /usr/local/lib/python3.7/site-packages/snowflake/connector/options.py:99:You have an incompatible version of 'pyarrow' installed (6.0.1), please install a version that adheres to: 'pyarrow<8.1.0,>=8.0.0; extra == "pandas"'
...
  ERROR! There were 1 warnings generated during the import

and

[13951](https://github.com/apache/airflow/runs/7520766090?check_suite_focus=true#step:10:13965)
tests/providers/google/common/auth_backend/test_google_openid.py F...... [ 74%]
...
>           assert "[email protected]" == current_user.email
[14171](https://github.com/apache/airflow/runs/7520766090?check_suite_focus=true#step:10:14185)
E           AttributeError: 'AnonymousUser' object has no attribute 'email'

Do not seem related to this PR for me.

[potiuk]

Yep. You modify dependencies so you also upgrade all others, and those errors are indeed happening in main (or were happening) - one of them I already fixed (the warning). The other I know what causes it and (new flask-login) and I am fixing it shortly. Bad luck - you attempted to update dependencies (via provider.yaml) when other dependencies broke main.

🤷

[Piatachock]

Oh, makes sense, thanks for the explanation. Please let me know if I can help in any way.
On the other hand, we at Yandex would be happy to pack this PR in next providers release; I would be on vacation for next week, but if we can help to make this happen - please let me or @peter-volkov know

[potiuk]

Just rebased it (I fixed both problem and merged the fix today). If it works and we merge it - it will get released.