Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement transfer for BigQuery native path S3/GCS #1850

Closed
wants to merge 12 commits into from

Commits on Mar 8, 2023

  1. Refactor SQLBaseOperator to reuse database & connection (#1833)

    Allow users to use `run_raw_sql` to convert from pandas dataframe
    created with `aql.dataframe` into a DuckDB in-memory table.
    
    Previously we were creating different database and connection instances
    within the base SQL operator unnecessarily.
    
    Fix: #1831
    tatiana authored Mar 8, 2023
    Configuration menu
    Copy the full SHA
    195ba59 View commit details
    Browse the repository at this point in the history

Commits on Mar 9, 2023

  1. Configuration menu
    Copy the full SHA
    5c56d20 View commit details
    Browse the repository at this point in the history

Commits on Mar 10, 2023

  1. Add basic Tiltfile for Python SDK local dev (#1819)

    https://tilt.dev/ is a powerful dev tool when using containers.
    
    It can automatically restart containers and sync files between localhost
    and container
    kaxil authored Mar 10, 2023
    Configuration menu
    Copy the full SHA
    ecf2d61 View commit details
    Browse the repository at this point in the history

Commits on Mar 14, 2023

  1. Fix RELEASE.md documentation broken link (#1834)

    Co-authored-by: Pankaj Koti <[email protected]>
    tatiana and pankajkoti authored Mar 14, 2023
    Configuration menu
    Copy the full SHA
    710cc61 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    a6206d2 View commit details
    Browse the repository at this point in the history
  3. Add argument skip_on_failure to CleanupOperator (#1837)

    # Description
    ## What is the current behavior?
    Currently, `astro.sql.cleanup` deletes temporary tables once upstream
    tasks are done, whether or not the DAG succeeded.
    
    While usually desirable, more detailed control is sometimes useful. For
    example, during DAG development, I may not want to keep regenerating all
    temporary tables while bugfixing failing tasks.
    
    ## What is the new behavior?
    
    - `CleanupOperator` has a new optional argument `skip_on_failure` that
    prevents table cleanup if any upstream task fails.
    - To mimic current behavior, `skip_on_failure=False` by default.
    
    This PR closes issue #1826. 
    
    ## Does this introduce a breaking change?
    No
    
    ### Checklist
    [x ] Created tests which fail without the change (if possible)
    [ ] Extended the README / documentation, if necessary
    scottleechua authored Mar 14, 2023
    Configuration menu
    Copy the full SHA
    456b2ba View commit details
    Browse the repository at this point in the history
  4. Fix CodeCov (#1815)

    # Description
    ## What is the current behavior?
    Currently, the codecov is broken because we don't generate the
    codecoverage.xml file for the entire set of test cases available, we
    generate it for only a subset of it like - PythonSDK or UTO or SQL CLI
    and codecov expects coverage file for all the tests cases, without which
    it leads to thinking the code coverage went down.
    
    There is a feature of flags and carry-forward flags -
    https://docs.codecov.com/docs/carryforward-flags and
    https://docs.codecov.com/docs/flags Which can help is carry forwarding
    the right coverage report from the correct commit if we generate the
    coverage report for only a subset of test cases.
    
    ## What is the new behavior?
    codecov has a feature of flags to handle the mono repo use case -
    `Flags`.
    
    
    ## Does this introduce a breaking change?
    Nope
    
    ### Checklist
    - [ ] Created tests which fail without the change (if possible)
    - [ ] Extended the README / documentation, if necessary
    utkarsharma2 authored Mar 14, 2023
    Configuration menu
    Copy the full SHA
    08b7367 View commit details
    Browse the repository at this point in the history
  5. Fix S3provider issue (#1846)

    # Description
    ## What is the current behavior?
    Fix s3 provider issue, needs {} instead of None in`transfer_config_args`
    config.
    utkarsharma2 authored Mar 14, 2023
    Configuration menu
    Copy the full SHA
    14bb7c3 View commit details
    Browse the repository at this point in the history

Commits on Mar 15, 2023

  1. Add example dags testcase as part of CI (#1845)

    # Description
    ## What is the current behavior?
    Add example dags as part of integration tests.
    utkarsharma2 authored Mar 15, 2023
    Configuration menu
    Copy the full SHA
    ba26711 View commit details
    Browse the repository at this point in the history

Commits on Mar 16, 2023

  1. Bump astro-runtime 7.4.0 version (#1848)

    Bump astro-runtime 7.4.0 version
    pankajastro authored Mar 16, 2023
    Configuration menu
    Copy the full SHA
    bf189de View commit details
    Browse the repository at this point in the history
  2. Implement transfer for BigQuery - read/write (#1829)

    **Please describe the feature you'd like to see**
    - Add `DataProvider` for Bigquery - read/write methods
    - Add non-native transfer implementation for GCS to BigQuery
    -  Add non-native transfer implementation for S3 to BigQuery
    - Add non-native transfer example DAG for BigQuery to Sqlite 
    - Add non-native transfer example DAG for BigQuery to Snowflake
    - Add example DAG
    - Add tests with 90% coverage
    
    **Acceptance Criteria**
    
    - [ ] All checks and tests in the CI should pass
    - [ ] Unit tests (90% code coverage or more, [once
    available](#191))
    - [ ] Integration tests (if the feature relates to a new database or
    external service)
    - [ ] Example DAG
    - [ ] Docstrings in
    [reStructuredText](https://peps.python.org/pep-0287/) for each of
    methods, classes, functions and module-level attributes (including
    Example DAG on how it should be used)
    - [ ] Exception handling in case of errors
    - [ ] Logging (are we exposing useful information to the user? e.g.
    source and destination)
    - [ ] Improve the documentation (README, Sphinx, and any other relevant)
    - [ ] How to use Guide for the feature
    ([example](https://airflow.apache.org/docs/apache-airflow-providers-postgres/stable/operators/postgres_operator_howto_guide.html))
    
    
    closes: #1732 
    closes: #1785
    closes: #1730
    
    ---------
    
    Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
    Co-authored-by: Utkarsh Sharma <[email protected]>
    Co-authored-by: Phani Kumar <[email protected]>
    4 people authored Mar 16, 2023
    Configuration menu
    Copy the full SHA
    d06d3e7 View commit details
    Browse the repository at this point in the history
  3. Add MySQL support (#1801)

    phanikumv authored Mar 16, 2023
    Configuration menu
    Copy the full SHA
    ee161d4 View commit details
    Browse the repository at this point in the history