-
Notifications
You must be signed in to change notification settings - Fork 43
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement transfer for BigQuery native path S3/GCS #1850
Commits on Mar 8, 2023
-
Refactor SQLBaseOperator to reuse database & connection (#1833)
Allow users to use `run_raw_sql` to convert from pandas dataframe created with `aql.dataframe` into a DuckDB in-memory table. Previously we were creating different database and connection instances within the base SQL operator unnecessarily. Fix: #1831
Configuration menu - View commit details
-
Copy full SHA for 195ba59 - Browse repository at this point
Copy the full SHA 195ba59View commit details
Commits on Mar 9, 2023
-
Configuration menu - View commit details
-
Copy full SHA for 5c56d20 - Browse repository at this point
Copy the full SHA 5c56d20View commit details
Commits on Mar 10, 2023
-
Add basic
Tiltfile
for Python SDK local dev (#1819)https://tilt.dev/ is a powerful dev tool when using containers. It can automatically restart containers and sync files between localhost and container
Configuration menu - View commit details
-
Copy full SHA for ecf2d61 - Browse repository at this point
Copy the full SHA ecf2d61View commit details
Commits on Mar 14, 2023
-
Fix RELEASE.md documentation broken link (#1834)
Co-authored-by: Pankaj Koti <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for 710cc61 - Browse repository at this point
Copy the full SHA 710cc61View commit details -
Configuration menu - View commit details
-
Copy full SHA for a6206d2 - Browse repository at this point
Copy the full SHA a6206d2View commit details -
Add argument
skip_on_failure
toCleanupOperator
(#1837)# Description ## What is the current behavior? Currently, `astro.sql.cleanup` deletes temporary tables once upstream tasks are done, whether or not the DAG succeeded. While usually desirable, more detailed control is sometimes useful. For example, during DAG development, I may not want to keep regenerating all temporary tables while bugfixing failing tasks. ## What is the new behavior? - `CleanupOperator` has a new optional argument `skip_on_failure` that prevents table cleanup if any upstream task fails. - To mimic current behavior, `skip_on_failure=False` by default. This PR closes issue #1826. ## Does this introduce a breaking change? No ### Checklist [x ] Created tests which fail without the change (if possible) [ ] Extended the README / documentation, if necessary
Configuration menu - View commit details
-
Copy full SHA for 456b2ba - Browse repository at this point
Copy the full SHA 456b2baView commit details -
# Description ## What is the current behavior? Currently, the codecov is broken because we don't generate the codecoverage.xml file for the entire set of test cases available, we generate it for only a subset of it like - PythonSDK or UTO or SQL CLI and codecov expects coverage file for all the tests cases, without which it leads to thinking the code coverage went down. There is a feature of flags and carry-forward flags - https://docs.codecov.com/docs/carryforward-flags and https://docs.codecov.com/docs/flags Which can help is carry forwarding the right coverage report from the correct commit if we generate the coverage report for only a subset of test cases. ## What is the new behavior? codecov has a feature of flags to handle the mono repo use case - `Flags`. ## Does this introduce a breaking change? Nope ### Checklist - [ ] Created tests which fail without the change (if possible) - [ ] Extended the README / documentation, if necessary
Configuration menu - View commit details
-
Copy full SHA for 08b7367 - Browse repository at this point
Copy the full SHA 08b7367View commit details -
# Description ## What is the current behavior? Fix s3 provider issue, needs {} instead of None in`transfer_config_args` config.
Configuration menu - View commit details
-
Copy full SHA for 14bb7c3 - Browse repository at this point
Copy the full SHA 14bb7c3View commit details
Commits on Mar 15, 2023
-
Add example dags testcase as part of CI (#1845)
# Description ## What is the current behavior? Add example dags as part of integration tests.
Configuration menu - View commit details
-
Copy full SHA for ba26711 - Browse repository at this point
Copy the full SHA ba26711View commit details
Commits on Mar 16, 2023
-
Bump astro-runtime 7.4.0 version (#1848)
Bump astro-runtime 7.4.0 version
Configuration menu - View commit details
-
Copy full SHA for bf189de - Browse repository at this point
Copy the full SHA bf189deView commit details -
Implement transfer for BigQuery - read/write (#1829)
**Please describe the feature you'd like to see** - Add `DataProvider` for Bigquery - read/write methods - Add non-native transfer implementation for GCS to BigQuery - Add non-native transfer implementation for S3 to BigQuery - Add non-native transfer example DAG for BigQuery to Sqlite - Add non-native transfer example DAG for BigQuery to Snowflake - Add example DAG - Add tests with 90% coverage **Acceptance Criteria** - [ ] All checks and tests in the CI should pass - [ ] Unit tests (90% code coverage or more, [once available](#191)) - [ ] Integration tests (if the feature relates to a new database or external service) - [ ] Example DAG - [ ] Docstrings in [reStructuredText](https://peps.python.org/pep-0287/) for each of methods, classes, functions and module-level attributes (including Example DAG on how it should be used) - [ ] Exception handling in case of errors - [ ] Logging (are we exposing useful information to the user? e.g. source and destination) - [ ] Improve the documentation (README, Sphinx, and any other relevant) - [ ] How to use Guide for the feature ([example](https://airflow.apache.org/docs/apache-airflow-providers-postgres/stable/operators/postgres_operator_howto_guide.html)) closes: #1732 closes: #1785 closes: #1730 --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Utkarsh Sharma <[email protected]> Co-authored-by: Phani Kumar <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for d06d3e7 - Browse repository at this point
Copy the full SHA d06d3e7View commit details -
Configuration menu - View commit details
-
Copy full SHA for ee161d4 - Browse repository at this point
Copy the full SHA ee161d4View commit details