Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add post endpoint for dataset events #37570

Merged

Conversation

edumuellerFSL
Copy link
Contributor

@edumuellerFSL edumuellerFSL commented Feb 20, 2024

related: #36723


This PR introduces a new POST endpoint for dataset events in Apache Airflow. It allows for the registration of dataset events, enabling more dynamic tracking and management of datasets within Airflow workflows. The changes include modifications to API endpoints, schema definitions, dataset managers, security permissions, and corresponding unit tests to ensure functionality and security compliance.

Changes

  • API Endpoints: Added a new post endpoint (post_dataset_event) to dataset_endpoint.py for creating dataset events, along with necessary security checks and action logging.
  • OpenAPI Specification: Updated the openapi/v1.yaml file to include the new post endpoint under /datasets/events, including request and response schema definitions.
  • Dataset Manager: Modified manager.py to support registering dataset changes without a task instance and to return a DatasetEvent object.
  • Security Permissions: Updated security_manager/override.py to include create permissions for the dataset resource.
  • UI and Types: Adjusted JavaScript components and API-generated types to accommodate the new changes.
  • Unit Tests: Added comprehensive tests in test_dataset_endpoint.py and test_dataset_schema.py to cover the new endpoint functionality and schema validation.

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@boring-cyborg boring-cyborg bot added area:API Airflow's REST/HTTP API area:providers area:UI Related to UI/UX. For Frontend Developers. area:webserver Webserver related Issues provider:fab labels Feb 20, 2024
@cmarteepants
Copy link
Collaborator

@Lee-W @pankajkoti

airflow/api_connexion/endpoints/dataset_endpoint.py Outdated Show resolved Hide resolved
airflow/datasets/manager.py Outdated Show resolved Hide resolved
@Lee-W Lee-W force-pushed the feature/add-dataset-event-post-endpoint branch 5 times, most recently from da64a59 to cbffbf9 Compare February 22, 2024 12:20
Copy link
Contributor

@ephraimbuddy ephraimbuddy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

airflow/api_connexion/endpoints/dataset_endpoint.py Outdated Show resolved Hide resolved
Copy link
Contributor

@dstandish dstandish left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

one or two suggestions

airflow/api_connexion/endpoints/dataset_endpoint.py Outdated Show resolved Hide resolved
@Lee-W Lee-W force-pushed the feature/add-dataset-event-post-endpoint branch 5 times, most recently from 664d1ec to 33feb95 Compare February 23, 2024 08:59
@Lee-W Lee-W force-pushed the feature/add-dataset-event-post-endpoint branch from 33feb95 to c0e6780 Compare February 25, 2024 22:54
@uranusjr uranusjr force-pushed the feature/add-dataset-event-post-endpoint branch from 12cd7a0 to 7f6d815 Compare February 26, 2024 07:01
@uranusjr uranusjr merged commit 9b17ff3 into apache:main Feb 26, 2024
19 of 58 checks passed
@ephraimbuddy ephraimbuddy added this to the Airflow 2.9.0 milestone Mar 6, 2024
@ephraimbuddy ephraimbuddy added the type:new-feature Changelog: New Features label Mar 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:API Airflow's REST/HTTP API area:providers area:UI Related to UI/UX. For Frontend Developers. area:webserver Webserver related Issues provider:fab type:new-feature Changelog: New Features
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants