Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deferrable mode for CreateBatchPredictionJobOperator #37818

Conversation

moiseenkov
Copy link
Contributor

Deferrable mode for the CreateBatchPredictionJobOperator is introduced.

@moiseenkov moiseenkov force-pushed the vertex-ai-create-batch-prediction-job-operator-deferrable branch from b1d38bf to 8efd4b6 Compare March 1, 2024 10:39
Copy link
Contributor

@dabla dabla left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice refactoring @moiseenkov, async operators are better as those are non-blocking and don't hold up workers while waiting for a response.

You could improve the tests like I did with my async msgraph operator here, so that you simulate the whole async execution cycle including the trigger execution. You'll see that I've create an execute_operator method, that way we simulate the whole behaviour in the same main thread and then we can do more than just simple verifications, we can actually also assert the result of that execution. Maybe we could move that execute_operator helper method somewhere to some central helper method in the tests package so it can be re-used across different tests for async operators. Doing so also allows you, like I did, to accumulate XCom interactions with the context. Don't know what the other Airflow project maintainers think of this approach.

@VladaZakharova
Copy link
Contributor

VladaZakharova commented Mar 4, 2024

Hi @eladkal @potiuk @pankajastro @hussein-awala !
Can we please merge these changes?

@potiuk potiuk merged commit ec220a8 into apache:main Mar 6, 2024
99 checks passed
@utkarsharma2 utkarsharma2 added this to the Airflow 2.9.0 milestone Mar 6, 2024
@ephraimbuddy ephraimbuddy added the changelog:skip Changes that should be skipped from the changelog (CI, tests, etc..) label Mar 6, 2024
potiuk added a commit to potiuk/airflow that referenced this pull request Mar 29, 2024
The apache#37818 bumped requirement for google-cloud-aiplatform to a
version that requires Pydantic so we need to remove it when
we are running "no-pydantic" tests.

The tests that import it should be skipped
when the dependency is removed.
potiuk added a commit that referenced this pull request Mar 29, 2024
…38617)

The #37818 bumped requirement for google-cloud-aiplatform to a
version that requires Pydantic so we need to remove it when
we are running "no-pydantic" tests.

The tests that import it should be skipped
when the dependency is removed.
ephraimbuddy pushed a commit that referenced this pull request Apr 2, 2024
…38617)

The #37818 bumped requirement for google-cloud-aiplatform to a
version that requires Pydantic so we need to remove it when
we are running "no-pydantic" tests.

The tests that import it should be skipped
when the dependency is removed.

(cherry picked from commit 69df34e)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:providers area:system-tests changelog:skip Changes that should be skipped from the changelog (CI, tests, etc..) kind:documentation provider:google Google (including GCP) related issues
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants