Release 1.4.0 #934
Merged
Release 1.4.0 #934
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
✅ Deploy Preview for sunny-pastelito-5ecb04 ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #934 +/- ##
=======================================
Coverage 95.72% 95.72%
=======================================
Files 59 59
Lines 2877 2877
=======================================
Hits 2754 2754
Misses 123 123 ☔ View full report in Codecov by Sentry. |
Merged
tatiana
added a commit
that referenced
this pull request
May 2, 2024
It seems the [google/re2](https://github.com/google/re2) release `1.1.20240501` is leading to an initialization issue in Netlify: ``` 2:32:59 PM: Running setup.py install for google-re2: finished with status 'error' 2:32:59 PM: ERROR: Command errored out with exit status 1: 2:32:59 PM: command: /opt/buildhome/python3.8/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"/tmp/pip-install-ti3twlby/google-re2/setup.py"'; __file__='"/tmp/pip-install-ti3twlby/google-re2/setup.py"';f=getattr(tokenize, '"open"', open)(__file__);code=f.read().replace('"rn"', '"n"');f.close();exec(compile(code, __file__, '"exec"'))' install --record /tmp/pip-record-8yg4w9lc/install-record.txt --single-version-externally-managed --compile --install-headers /opt/buildhome/python3.8/include/site/python3.8/google-re2 2:32:59 PM: cwd: /tmp/pip-install-ti3twlby/google-re2/ 2:32:59 PM: Complete output (28 lines): 2:32:59 PM: running install 2:32:59 PM: /opt/buildhome/python3.8/lib/python3.8/site-packages/setuptools/_distutils/cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated. 2:32:59 PM: git checkout main 2:32:59 PM: ******************************************************************************** 2:32:59 PM: Please avoid running setup.py directly. 2:32:59 PM: Instead, use pypa/build, pypa/installer or other 2:32:59 PM: standards-based tools. 2:32:59 PM: See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details. 2:32:59 PM: ******************************************************************************** 2:32:59 PM: git checkout main 2:32:59 PM: self.initialize_options() 2:32:59 PM: running build 2:32:59 PM: running build_py 2:32:59 PM: creating build 2:32:59 PM: creating build/lib.linux-x86_64-cpython-38 2:32:59 PM: copying re2.py -> build/lib.linux-x86_64-cpython-38 2:32:59 PM: running build_ext 2:32:59 PM: building '_re2' extension 2:32:59 PM: creating build/temp.linux-x86_64-cpython-38 2:32:59 PM: x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -I/opt/buildhome/python3.8/lib/python3.8/site-packages/pybind11/include -I/opt/buildhome/python3.8/include -I/usr/include/python3.8 -c _re2.cc -o build/temp.linux-x86_64-cpython-38/_re2.o -fvisibility=hidden 2:32:59 PM: _re2.cc:13:10: fatal error: absl/strings/string_view.h: No such file or directory 2:32:59 PM: 13 | #include absl/strings/string_view.h 2:32:59 PM: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2:32:59 PM: compilation terminated. 2:32:59 PM: error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1 2:32:59 PM: ---------------------------------------- 2:33:00 PM: Failed during stage 'Install dependencies': dependency_installation script returned non-zero exit code: 1 2:33:00 PM: ERROR: Command errored out with exit status 1: /opt/buildhome/python3.8/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"/tmp/pip-install-ti3twlby/google-re2/setup.py"'; __file__='"/tmp/pip-install-ti3twlby/google-re2/setup.py"';f=getattr(tokenize, '"open"', open)(__file__);code=f.read().replace('"rn"', '"n"');f.close();exec(compile(code, __file__, '"exec"'))' install --record /tmp/pip-record-8yg4w9lc/install-record.txt --single-version-externally-managed --compile --install-headers /opt/buildhome/python3.8/include/site/python3.8/google-re2 Check the logs for full command output. ``` As seen in: https://app.netlify.com/sites/sunny-pastelito-5ecb04/deploys/663395dcce3dc1000843d9fd During the Cosmos release 1.4.0: #934
dosubot
bot
added
the
size:S
dosubot
bot
added
area:dependencies
|
We need to address at least two issues found while validating 1.4.0a2 before we do the stable release: (1) Make sure we're able to change https://github.com/astronomer/cosmos-demo/blob/main/dags/basic/simple_dag.py to use (2) Make sure Cosmos will not raise the following exception when running Links to the issues: |
|
After solving the previously observed issues, we found two new topics: For (1), for now, we improved the log message in Cosmos - and we're fixing the root issue in Airflow itself. |
pankajkoti
approved these changes
May 13, 2024
There was a problem hiding this comment.
Do we need to rebase to include recent fixes including #954 as a commit in the tree?
Issue: pypa/twine#1059 Upgrade GH action to support Metadata-Version: 2.3: https://github.com/pypa/gh-action-pypi-publish/releases/tag/v1.8.13
Faced while running the following command in the CI (I was not able to reproduce this locally):
```
Run python -m build
```
Resulted in the stacktrace:
```
val = self.func(instance)
File "/tmp/build-env-o83v4k67/lib/python3.10/site-packages/hatchling/builders/wheel.py", line 247, in default_file_selection_options
raise ValueError(message)
ValueError: Unable to determine which files to ship inside the wheel using the following heuristics: https://hatch.pypa.io/latest/plugins/builder/wheel/#default-file-selection
The most likely cause of this is that there is no directory that matches the name of your project (astronomer_cosmos).
At least one file selection option must be defined in the `tool.hatch.build.targets.wheel` table, see: https://hatch.pypa.io/latest/config/build/
As an example, if you intend to ship a directory named `foo` that resides within a `src` directory located at the root of your project, you can define the following:
[tool.hatch.build.targets.wheel]
packages = ["src/foo"]
ERROR Backend subprocess exited when trying to invoke build_wheel
```
As seen in:
https://github.com/astronomer/astronomer-cosmos/actions/runs/8986616060/job/24683237685
arojasb3
pushed a commit
to arojasb3/astronomer-cosmos
that referenced
this pull request
Jul 14, 2024
It seems the [google/re2](https://github.com/google/re2) release `1.1.20240501` is leading to an initialization issue in Netlify: ``` 2:32:59 PM: Running setup.py install for google-re2: finished with status 'error' 2:32:59 PM: ERROR: Command errored out with exit status 1: 2:32:59 PM: command: /opt/buildhome/python3.8/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"/tmp/pip-install-ti3twlby/google-re2/setup.py"'; __file__='"/tmp/pip-install-ti3twlby/google-re2/setup.py"';f=getattr(tokenize, '"open"', open)(__file__);code=f.read().replace('"rn"', '"n"');f.close();exec(compile(code, __file__, '"exec"'))' install --record /tmp/pip-record-8yg4w9lc/install-record.txt --single-version-externally-managed --compile --install-headers /opt/buildhome/python3.8/include/site/python3.8/google-re2 2:32:59 PM: cwd: /tmp/pip-install-ti3twlby/google-re2/ 2:32:59 PM: Complete output (28 lines): 2:32:59 PM: running install 2:32:59 PM: /opt/buildhome/python3.8/lib/python3.8/site-packages/setuptools/_distutils/cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated. 2:32:59 PM: git checkout main 2:32:59 PM: ******************************************************************************** 2:32:59 PM: Please avoid running setup.py directly. 2:32:59 PM: Instead, use pypa/build, pypa/installer or other 2:32:59 PM: standards-based tools. 2:32:59 PM: See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details. 2:32:59 PM: ******************************************************************************** 2:32:59 PM: git checkout main 2:32:59 PM: self.initialize_options() 2:32:59 PM: running build 2:32:59 PM: running build_py 2:32:59 PM: creating build 2:32:59 PM: creating build/lib.linux-x86_64-cpython-38 2:32:59 PM: copying re2.py -> build/lib.linux-x86_64-cpython-38 2:32:59 PM: running build_ext 2:32:59 PM: building '_re2' extension 2:32:59 PM: creating build/temp.linux-x86_64-cpython-38 2:32:59 PM: x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -I/opt/buildhome/python3.8/lib/python3.8/site-packages/pybind11/include -I/opt/buildhome/python3.8/include -I/usr/include/python3.8 -c _re2.cc -o build/temp.linux-x86_64-cpython-38/_re2.o -fvisibility=hidden 2:32:59 PM: _re2.cc:13:10: fatal error: absl/strings/string_view.h: No such file or directory 2:32:59 PM: 13 | #include absl/strings/string_view.h 2:32:59 PM: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2:32:59 PM: compilation terminated. 2:32:59 PM: error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1 2:32:59 PM: ---------------------------------------- 2:33:00 PM: Failed during stage 'Install dependencies': dependency_installation script returned non-zero exit code: 1 2:33:00 PM: ERROR: Command errored out with exit status 1: /opt/buildhome/python3.8/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"/tmp/pip-install-ti3twlby/google-re2/setup.py"'; __file__='"/tmp/pip-install-ti3twlby/google-re2/setup.py"';f=getattr(tokenize, '"open"', open)(__file__);code=f.read().replace('"rn"', '"n"');f.close();exec(compile(code, __file__, '"exec"'))' install --record /tmp/pip-record-8yg4w9lc/install-record.txt --single-version-externally-managed --compile --install-headers /opt/buildhome/python3.8/include/site/python3.8/google-re2 Check the logs for full command output. ``` As seen in: https://app.netlify.com/sites/sunny-pastelito-5ecb04/deploys/663395dcce3dc1000843d9fd During the Cosmos release 1.4.0: astronomer#934
arojasb3
pushed a commit
to arojasb3/astronomer-cosmos
that referenced
this pull request
Jul 14, 2024
Features * Add dbt docs natively in Airflow via plugin by @dwreeves in astronomer#737 * Add support for ``InvocationMode.DBT_RUNNER`` for local execution mode by @jbandoro in astronomer#850 * Support partial parsing to render DAGs faster when using ``ExecutionMode.LOCAL``, ``ExecutionMode.VIRTUALENV`` and ``LoadMode.DBT_LS`` by @dwreeves in astronomer#800 * Improve performance by 22-35% or more by caching partial parse artefact by @tatiana in astronomer#904 * Add Azure Container Instance as Execution Mode by @danielvdende in astronomer#771 * Add dbt build operators by @dylanharper-qz in astronomer#795 * Add dbt profile config variables to mapped profile by @ykuc in astronomer#794 * Add more template fields to ``DbtBaseOperator`` by @dwreeves in astronomer#786 * Add ``pip_install_options`` argument to operators by @octiva in astronomer#808 Bug fixes * Make ``PostgresUserPasswordProfileMapping`` schema argument optional by @FouziaTariq in astronomer#683 * Fix ``folder_dir`` not showing on logs for ``DbtDocsS3LocalOperator`` by @PrimOox in astronomer#856 * Improve ``dbt ls`` parsing resilience to missing tags/config by @tatiana in astronomer#859 * Fix ``operator_args`` modified in place in Airflow converter by @jbandoro in astronomer#835 * Fix Docker and Kubernetes operators execute method resolution by @jbandoro in astronomer#849 * Fix ``TrinoBaseProfileMapping`` required parameter for non method authentication by @AlexandrKhabarov in astronomer#921 * Fix global flags for lists by @ms32035 in astronomer#863 * Fix ``GoogleCloudServiceAccountDictProfileMapping`` when getting values from the Airflow connection ``extra__`` keys by @glebkrapivin in astronomer#923 * Fix using the dag as a keyword argument as ``specific_args_keys`` in DbtTaskGroup by @tboutaour in astronomer#916 * Fix ACI integration (``DbtAzureContainerInstanceBaseOperator``) by @danielvdende in astronomer#872 * Fix setting dbt project dir to the tmp dir by @dwreeves in astronomer#873 * Fix dbt docs operator to not use ``graph.gpickle`` file when ``--no-write-json`` is passed by @dwreeves in astronomer#883 * Make Pydantic a required dependency by @pankajkoti in astronomer#939 * Gracefully error if users try to ``emit_datasets`` with ``Airflow 2.9.0`` or ``2.9.1`` by @tatiana in astronomer#948 * Fix parsing tests that have no parents in astronomer#933 by @jlaneve * Correct ``root_path`` in partial parse cache by @pankajkoti in astronomer#950 Docs * Fix docs homepage link by @jlaneve in astronomer#860 * Fix docs ``ExecutionConfig.dbt_project_path`` by @jbandoro in astronomer#847 * Fix typo in MWAA getting started guide by @jlaneve in astronomer#846 * Fix typo related to exporting docs to GCS by @tboutaour in astronomer#922 * Improve partial parsing docs by @tatiana in astronomer#898 * Improve docs for datasets for airflow >= 2.4 by @SiddiqueAhmad in astronomer#879 * Improve test behaviour docs to highlight ``warning`` feature in the ``virtualenv`` mode by @mc51 in astronomer#910 * Fix docs typo by @SiddiqueAhmad in astronomer#917 * Improve Astro docs by @RNHTTR in astronomer#951 Others * Add performance integration tests by @jlaneve in astronomer#827 * Enable ``append_env`` in ``operator_args`` by default by @tatiana in astronomer#899 * Change default ``append_env`` behaviour depending on Cosmos ``ExecutionMode`` by @pankajkoti and @pankajastro in astronomer#954 * Expose the ``dbt`` graph in the ``DbtToAirflowConverter`` class by @tommyjxl in astronomer#886 * Improve dbt docs plugin rendering padding by @dwreeves in astronomer#876 * Add ``connect_retries`` to databricks profile to fix expensive integration failures by @jbandoro in astronomer#826 * Add import sorting (isort) to Cosmos by @jbandoro in astronomer#866 * Add Python 3.11 to CI/tests by @tatiana and @jbandoro in astronomer#821, astronomer#824 and astronomer#825 * Fix failing ``test_created_pod`` for ``apache-airflow-providers-cncf-kubernetes`` after v8.0.0 update by @jbandoro in astronomer#854 * Extend ``DatabricksTokenProfileMapping`` test to include session properties by @tatiana in astronomer#858 * Fix broken integration test uncovered from Pytest 8.0 update by @jbandoro in astronomer#845 * Add Apache Airflow 2.9 to the test matrix by @tatiana in astronomer#940 * Replace deprecated ``DummyOperator`` by ``EmptyOperator`` if Airflow >=2.4.0 by @tatiana in astronomer#900 * Improve logs to troubleshoot issue in 1.4.0a2 with astro-cli by @tatiana in astronomer#947 * Fix issue when publishing a new release to PyPI by @tatiana in astronomer#946 * Pre-commit hook updates in astronomer#820, astronomer#834, astronomer#843 and astronomer#852, astronomer#890, astronomer#896, astronomer#901, astronomer#905, astronomer#908, astronomer#919, astronomer#931, astronomer#941
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Features
InvocationMode.DBT_RUNNERfor local execution mode by @jbandoro in Add support forInvocationMode.DBT_RUNNERfor local execution mode #850ExecutionMode.LOCAL,ExecutionMode.VIRTUALENVandLoadMode.DBT_LSby @dwreeves in Support partial parsing #800DbtBaseOperatorby @dwreeves in Add more template fields toDbtBaseOperator#786pip_install_optionsargument to operators by @octiva in feat: add pip_install_options as an operator arg #808Bug fixes
PostgresUserPasswordProfileMappingschema argument optional by @FouziaTariq in Make PostgresUserPasswordProfileMapping schema profile argument schem… #683folder_dirnot showing on logs forDbtDocsS3LocalOperatorby @PrimOox in fixfolder_dirnot showing on logs forDbtDocsS3LocalOperator#856dbt lsparsing resilience to missing tags/config by @tatiana in Improvedbt lsparsing resilience to missing tags/config #859operator_argsmodified in place in Airflow converter by @jbandoro in Fix:operator_argsmodified in place in Airflow converter #835TrinoBaseProfileMappingrequired parameter for non method authentication by @AlexandrKhabarov in fix: missed required parameter for non method authentication #921GoogleCloudServiceAccountDictProfileMappingwhen getting values from the Airflow connectionextra__keys by @glebkrapivin in Fixed getting values from extra__ keys in airflow con #923specific_args_keysin DbtTaskGroup by @tboutaour in Fix instantiatingDbtTaskGroupwith argumentdag(outside ofDAGcontext) #916DbtAzureContainerInstanceBaseOperator) by @danielvdende in Fix ACI integration #872graph.gpicklefile when--no-write-jsonis passed by @dwreeves in [Bugfix] Dbt docs operator should not look forgraph.gpicklefile when--no-write-jsonis passed. #883emit_datasetswithAirflow 2.9.0or2.9.1by @tatiana in Gracefully error if users try to emit_datasets with Airflow 2.9.0 or 2.9.1 #948root_pathin partial parse cache by @pankajkoti in Correct staleroot_pathin partial parse file #950Docs
ExecutionConfig.dbt_project_pathby @jbandoro in Fix docs:ExecutionConfig.dbt_project_path#847warningfeature in thevirtualenvmode by @mc51 in update testing behavior docs #910Others
append_envinoperator_argsby default by @tatiana in Enableappend_envinoperator_argsby default #899append_envbehaviour depending on CosmosExecutionModeby @pankajkoti and @pankajastro in Set default value forappend_envbased on execution modes #954dbtgraph in theDbtToAirflowConverterclass by @tommyjxl in Expose the dbt graph in theDbtToAirflowConverterclass #886connect_retriesto databricks profile to fix expensive integration failures by @jbandoro in Addconnect_retriesto databricks profile to fix expensive integration failures #826test_created_podwith Py3.11 #825test_created_podforapache-airflow-providers-cncf-kubernetesafter v8.0.0 update by @jbandoro in Fix failing test_created_pod forapache-airflow-providers-cncf-kubernetesafter v8.0.0 update #854DatabricksTokenProfileMappingtest to include session properties by @tatiana in ExtendDatabricksTokenProfileMappingtest to include session properties #858DummyOperatorbyEmptyOperatorif Airflow >=2.4.0 by @tatiana in Replace deprecated DummyOperator by EmptyOperator if Airflow >=2.4.0 #900