[CT-2672] [Bug] Cannot import name 'Unpack' from 'typing_extensions' (dbt-core 1.5.1) #7828
Comments
github-actions
bot
changed the title
|
That is awesome that you are tying out programmatic invocations @Almaz-KG 🤩 ! I didn't try out with the Reprex
This worked for me: dbt run --select tag:my_tagAnd so did this: python runner.pyOuput: 17:43:23 Running with dbt=1.5.0
17:43:23 Found 1 model, 0 tests, 0 snapshots, 0 analyses, 307 macros, 0 operations, 0 seed files, 0 sources, 0 exposures, 0 metrics, 0 groups
17:43:23
17:43:23 Concurrency: 5 threads (target='postgres')
17:43:23
17:43:23 1 of 1 START sql view model dbt_dbeatty.my_model ............................... [RUN]
17:43:23 1 of 1 OK created sql view model dbt_dbeatty.my_model .......................... [CREATE VIEW in 0.13s]
17:43:23
17:43:23 Finished running 1 view model in 0 hours 0 minutes and 0.35 seconds (0.35s).
17:43:23
17:43:23 Completed successfully
17:43:23
17:43:23 Done. PASS=1 WARN=0 ERROR=0 SKIP=0 TOTAL=1
my_model: success |
Oh, I've come across something like this after typing-extensions upgrade and made a fix in this commit: It will be included in the upcoming release this or next week. |
|
I think it's worth to exclude only those versions that break something in dbt-core. Not taking into account the transition to major version 3, I remember that 3.0.2 and 3.1.0 were problematic, so they are candidates for exclusion. On the other hand if you rely on some functionality that exists only from a version X then you should obviously pin that version as the minimal one and remove all exclusions before X as redundant: If I were in your place, I would be looking at pull requests from dependabot and either: The option "a" is better for users because it makes it easier to use dbt-core in combination with other packages that may also depend on mashumaro. But if dbt-core is positioned as the product rather than as a library than option "b" or even strict pin to the specific version of a package is also a good choice. |
|
Hey @Fatal1ty ! WDYT, should I wait for the fixes in dbt-core or can I hack this dependency issue by myself? |
Forget what I wrote before. What I encountered is unrelated to the current issue. Has anyone else tried to reproduce this issue? Because I can't: @Almaz-KG It looks like you failed to do imports on the first line in your environment:
I bet you have typing-extensions<4.1.0 installed. |
No, I have version 4.6.3. The full list of deps with their versions (got from I've also played today with docker images (with different python versions, 3.9, 3.9.5, 3.10), and can't reproduce the issue. But, I can reproduce it in databricks cluster. IDK where is the difference comes in |
|
I've also tried to reproduce this, but have not been able to. |
|
On dbt-core==1.6.1 and dbt-core==1.6.2 I get something similar (on databricks): Fixed by adding typing_extensions==4.7.1 as a dependency |
|
I've also received the same issue with typing extensions, adding 4.7.1 and it seems to work again but isn't preferable. Should only need to pin a dependency to dbt-databricks not also typing_extensions. |
I got the same issue when running on Databricks Runtime 13.3 LTS. But on Databricks Runtime 12.2 LTS it worked out fine. I did not try to fix This was tried with dbt-core 1.6.2 and dbt-databricks 1.6.4 + 1.6.5. Here my full stack trace for further investigation for the devs: Hope that helps to fix this issue. |
|
Hi, any updates on this issue? We are running into the same thing. We are implementing Programmatic Invocation here as well and have big plans for it! To isolate this problem I use this small bit of Python code and simply run it from the Databricks Workspace on a cluster that only has dbt-databricks 1.6.6 installed: print("Starting")
from dbt.cli.main import dbtRunner, dbtRunnerResult
dbt = dbtRunner()
cli_args = ['--version']
res: dbtRunnerResult = dbt.invoke(cli_args)
print("The End")I have tested it with different LTS Databricks runtime versions (12.2 and 13.3), different Access modes (Shared and Single) and with or without installing typing_extenstions 4.7.1. All using dbt-databricks 1.6.6. These are the results:
My complete stack trace when I get the error looks like this: ImportError: cannot import name 'override' from 'typing_extensions' (/databricks/python/lib/python3.10/site-packages/typing_extensions.py)
---------------------------------------------------------------------------
ImportError Traceback (most recent call last)
File <command-824890481590461>, line 3
1 print("Starting")
----> 3 from dbt.cli.main import dbtRunner, dbtRunnerResult
5 dbt = dbtRunner()
6 cli_args = ['--version']
File /local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/dbt/cli/__init__.py:1
----> 1 from .main import cli as dbt_cli # noqa
File /local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/dbt/cli/main.py:14
6 import click
7 from click.exceptions import (
8 Exit as ClickExit,
9 BadOptionUsage,
10 NoSuchOption,
11 UsageError,
12 )
---> 14 from dbt.cli import requires, params as p
15 from dbt.cli.exceptions import (
16 DbtInternalException,
17 DbtUsageException,
18 )
19 from dbt.contracts.graph.manifest import Manifest
File /local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/dbt/cli/requires.py:3
1 import dbt.tracking
2 from dbt.version import installed as installed_version
----> 3 from dbt.adapters.factory import adapter_management, register_adapter
4 from dbt.flags import set_flags, get_flag_dict
5 from dbt.cli.exceptions import (
6 ExceptionExit,
7 ResultExit,
8 )
File /local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/dbt/adapters/factory.py:8
5 from pathlib import Path
6 from typing import Any, Dict, List, Optional, Set, Type
----> 8 from dbt.adapters.base.plugin import AdapterPlugin
9 from dbt.adapters.protocol import AdapterConfig, AdapterProtocol, RelationProtocol
10 from dbt.contracts.connection import AdapterRequiredConfig, Credentials
File /local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/dbt/adapters/base/__init__.py:6
4 from dbt.contracts.connection import Credentials # noqa: F401
5 from dbt.adapters.base.meta import available # noqa: F401
----> 6 from dbt.adapters.base.connections import BaseConnectionManager # noqa: F401
7 from dbt.adapters.base.relation import ( # noqa: F401
8 BaseRelation,
9 RelationType,
10 SchemaSearchMap,
11 )
12 from dbt.adapters.base.column import Column # noqa: F401
File /local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/dbt/adapters/base/connections.py:35
26 import dbt.exceptions
27 from dbt.contracts.connection import (
28 Connection,
29 Identifier,
(...)
33 AdapterResponse,
34 )
---> 35 from dbt.contracts.graph.manifest import Manifest
36 from dbt.adapters.base.query_headers import (
37 MacroQueryStringSetter,
38 )
39 from dbt.events import AdapterLogger
File /local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/dbt/contracts/graph/manifest.py:27
24 from typing_extensions import Protocol
25 from uuid import UUID
---> 27 from dbt.contracts.graph.nodes import (
28 BaseNode,
29 Documentation,
30 Exposure,
31 GenericTestNode,
32 GraphMemberNode,
33 Group,
34 Macro,
35 ManifestNode,
36 Metric,
37 ModelNode,
38 DeferRelation,
39 ResultNode,
40 SemanticModel,
41 SourceDefinition,
42 UnpatchedSourceDefinition,
43 )
44 from dbt.contracts.graph.unparsed import SourcePatch, NodeVersion, UnparsedVersion
45 from dbt.contracts.graph.manifest_upgrade import upgrade_manifest_json
File /local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/dbt/contracts/graph/nodes.py:61
59 from dbt_semantic_interfaces.references import MetricReference as DSIMetricReference
60 from dbt_semantic_interfaces.type_enums import MetricType, TimeGranularity
---> 61 from dbt_semantic_interfaces.parsing.where_filter_parser import WhereFilterParser
63 from .model_config import (
64 NodeConfig,
65 SeedConfig,
(...)
72 SemanticModelConfig,
73 )
76 # =====================================================================
77 # This contains the classes for all of the nodes and node-like objects
78 # in the manifest. In the "nodes" dictionary of the manifest we find
(...)
95 # Various parent classes and node attribute classes
96 # ==================================================
File /local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/dbt_semantic_interfaces/parsing/where_filter_parser.py:14
7 from dbt_semantic_interfaces.call_parameter_sets import (
8 FilterCallParameterSets,
9 ParseWhereFilterException,
10 )
11 from dbt_semantic_interfaces.parsing.where_filter.parameter_set_factory import (
12 ParameterSetFactory,
13 )
---> 14 from dbt_semantic_interfaces.parsing.where_filter.where_filter_dimension import (
15 WhereFilterDimensionFactory,
16 )
17 from dbt_semantic_interfaces.parsing.where_filter.where_filter_entity import (
18 WhereFilterEntityFactory,
19 )
20 from dbt_semantic_interfaces.parsing.where_filter.where_filter_time_dimension import (
21 WhereFilterTimeDimensionFactory,
22 )
File /local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/dbt_semantic_interfaces/parsing/where_filter/where_filter_dimension.py:5
1 from __future__ import annotations
3 from typing import List, Optional, Sequence
----> 5 from typing_extensions import override
7 from dbt_semantic_interfaces.errors import InvalidQuerySyntax
8 from dbt_semantic_interfaces.protocols.protocol_hint import ProtocolHint |
|
@thijs-nijhuis-shell I guess it would be helpful if you could find out for each runtime / mode what version of package |
But shouldn't dbt-core depend on the right typing_extensions as it appearently needs it? |
I think I found something interesting when trying to fetch the versions. First off, there is no difference in type_extensions or python version between Single or Shared clusters. And, obviously, in my runs where I pre-install 4.7.1, that is the type_extensions version pip returns as well. So the only two situation to distinguish are runtime 12.2 vs 13.3. However, what does make a difference is whether you have dbt-core (or dbt-databricks, same result) pre-installed or not. These are the results:
So on 12.2, installing dbt-core also installs a new version op type_extensions. On 13.3 it does not. My guess is that somewhere in the dependencies of dbt-core (or any of its packages) it says type_extensions >= 4.2 or something like that triggering an update on 12.2 but not 13.3. Does that make sense? BTW, when I look at the requirements of dbt-core itself it says >=3.7.4. So maybe one of the other packages it uses caused the upgrade to 4.8.0 on 12.2. |
|
Any idea on how to do that? |
|
Can you try to check it with pipdeptree? https://pypi.org/project/pipdeptree/ As workaround maybe it works to force pip to install the 4.8.0 version of typing_extensions after you installed dbt-core / dbt-databricks. |
|
Hey @thijs-nijhuis-shell, I don't know your setup in Databricks, but if you're using code inside dbx notebooks, please also try this command before running any dbt-api call (restarting python process) # Restart python process to ensure new installed packages are used
dbutils.library.restartPython()TBH, dunno why, but it's required if you're installing deps inside the notebook |
|
The root cause of this issue comes from dbt-semantic-interfaces: |
|
@Almaz-Murzabekov I guess this should not be required when you install packages with the %pip magic or the init cluster script. |
@Fatal1ty thank you for opening the PR on dbt-semantic-interfaces! I want to be clear on what the PR in dbt-semantic-interfaces will be resolving though. It won't resolve the original issue that was seen in dbt-core 1.5 because dbt-semantic-interfaces wasn't in dbt-core 1.5. However it should help resolve the similar issue seen in 1.6 brought up in September. |
|
As @QMalcolm is calling out, it appears that there are two similar error messages with separate root causes and fixes:
Based on #7828 (comment), it sounds like the 2nd one will be fixed by dbt-labs/dbt-semantic-interfaces#193 Based on #7828 (comment), it originally sounded like the 1st one could be solved by a different version of So if I'm reading things correctly, we have a fix for dbt-core 1.6+, but do not yet have a path forward for the original issue reported for 1.5. |
|
I'm not convinced it will fully fix 1.6+ either unfortunately 😞 dbt-semantic-interfaces wasn't previously excluding |
|
@QMalcolm and I have done some further digging, and this comment #7828 (comment) was extremely helpful in our investigation, so big thanks to @thijs-nijhuis-shell What appears to be happening is Databricks in particular is configuring its initial python environment with typing-extensions set to 4.1/4.3, and a subsequent install of dbt-core 1.6 will not update this because 1.6 is set to require ~=4.0 (by way of dbt-semantic-interfaces). Since everything works if you install dbt-core (or typing-extensions 4.7.1) first, this suggests databricks is actually perfectly happy with 4.7 or whatever. However, that databricks 13.3 overwrites the typing-extensions module with 4.3 (while 12.2 leaves that well enough alone) is a bit of a cause for concern. Like, why would it do that? We're ready to backport the dbt-semantic-interfaces change to address the issue with 1.6, but we'd like to confirm that Databricks 13.3 is happy with a later version of typing_extensions before we merge and deploy. We'll update here once we're done. |
|
Worth noting - there is nothing I can find in dbt-core's existing dependency tree requiring typing-extensions > 4.1.0 OR restricting it to < 4.4. Mashumaro sets the current floor in production at >=4.1.0, so if you install databricks first there's no real reason for pip to update typing-extensions, and if you install dbt-core first there's no real reason for databricks to clobber typing-extensions on 13.3 but not on 12.2. |
|
I think it would make sence then to move this issue over to the Databricks project, or? |
|
I can file a ticket; I work on the adapter specifically, so I don't have a lot of insight into the DBR. |
|
The databricks 13.3 runtime appears happy with typing_extensions >= 4.3, so we will merge the relevant fixes into the dbt-semantic-interfaces package today, with plans to deploy the backport tomorrow. This won't affect existing installations of dbt-core or dbt-databricks, as an update to typing-extensions will not automatically trigger, but new installations should update versions accordingly. Worth noting, the databricks runtime install might still clobber the typing-extensions version on pre-installed dbt-databricks packages, so it could be necessary to install/update dbt-databricks (or even typing-extensions) after the runtime has spun up. I'll update here when the backport is deployed, as that should address most issues people are encountering with dbt-core 1.6. |
Thanks! I will make sure to test it on our end when available. |
|
dbt-semantic-interfaces 0.2.3 has been deployed: https://pypi.org/project/dbt-semantic-interfaces/0.2.3/ At this point, doing a pip upgrade (for existing installations) or doing a fresh install of dbt-core 1.6 or a corresponding adapter package should pin the typing-extensions version to something past 4.4.0. As mentioned earlier, this will NOT fix issues with 1.5 installs, and even on 1.6 any installation that clobbers the typing_extensions version may still cause problems, but at least if you install dbt-core it should now update typing-extensions appropriately. Thanks to @Fatal1ty for submitting the quick patch on the DSI side and everyone here (and especially @thijs-nijhuis-shell ) for the detailed info on what was broken. |
|
@tlento, first off, big thanks for the quick fix! The good news is that the error is indeed gone! So now, when running on DBX 13.3, I can run my test script without any errors and without installing any typing-extensions version. When I check the typing-extensions after installing dbt-core 1.6.6 it says 4.8.0, just like on DBX 12.2. I don't get these warning when running on DBX 12.2. To check the package versioning, I ran the pipdeptree tool that @leo-schick suggested on both 12.2 and 13.3 after installing dbt-databricks 1.6.6. The first list below is of 12.2 and the second for 13.3. I marked each row that uses a different versions with a * at the start in both lists. Hope this might help someone spot something odd. 12.2 deptreelist 13.3 deptreelist |
|
@thijs-nijhuis-shell The warnings What’s New In Python 3.10 — Python 3.10.13 documentation
|
|
@thijs-nijhuis-shell if it's always ImportHookFinder that may be due to the newrelic python agent. Upgrading their package may resolve it. newrelic/newrelic-python-agent#436 It doesn't appear that dbt relies on this anywhere, as a clean install of dbt-databricks does not bundle that package, nor can I find any evidence of it (or the ImportHookFinder) being referenced in vendored code within my venv (although I admit I'm not very good at that sort of investigation). |
|
Hi @Hangzhi and @tlento, thanks for the feedback. First off, I ran my sample script on the newly release dbt-core 1.7.0 today but I got the old 'typing_extension' error on DBX 13.3. So I think fix didn't make it into that release. I don't have any issues when using dbt-core 1.6.6. Second, the warnings. As you are probably aware, I am not a seasoned python programmer by any standard so I am struggling a bit on debugging this one. So I'll just write down what I tried (some of which might not make any sense at all) and hope this helps a more knowledgeable person think of something.
I think I am going to implement this workaround for now but hope we can find a proper fix for this at some point: with warnings.catch_warnings():
warnings.simplefilter("error", ImportWarning)
try:
from dbt.cli.main import dbtRunner, dbtRunnerResult
except ImportWarning:
pass
except:
raiseEdit: This workaround doesn't work as this will stop importing the dbt package. See comments below. |
🤦 sorry, I forgot to backport and deploy the change to dbt-semantic-interfaces to the 0.4 branch so 1.7.0 is still using the old dependency. The fix is up in dbt-labs/dbt-semantic-interfaces#205 and should be deployed early next week. Regarding the warnings, I can't repro them at all on a local install of dbt-databricks~=1.6 into a clean virtual environment, so I suspect they have something to do with either the databricks 13.3 runtime configuration itself, or the specific python environment setup you're using there. Note if it is due to |
|
The fix for dbt-semantic-interfaces has been backported to the 0.4.latest branch and deployed as v0.4.1. This should address the import issues described here for dbt-core 1.7.x, although a clean reinstall or manual upgrade of the dbt-semantic-interfaces package might be necessary. The problem originally raised in this issue does not appear to be due to a dependency mismatch, as the I believe the problem was once again with the way databricks runtime installs packages, which may lead it to clobber existing installs back to version pins in the requirements defined for the base dbx runtime environment. Ensuring that dbt is installed after the databricks runtime finishes initializing should now solve all of these issues. I've added a PR to update the dbt-core dependency to reflect the reality that dbt-core now requires typing-extensions 4.4 or later. Once that merges I think it's safe to consider this issue closed, unless someone here runs into a new version of this same problem. |
|
@tlento , thanks for fixing it for 1.7.x as well and all the effort you put into this. Unfortunately, my workaround to suppress the warnings doesn't work after all. When promoting the warnings to errors the dbt library is not loaded (completely) which makes sense of course. For some reason, when I set the warnings to 'ignore' instead of error like the python documentation says the warnings are still shown. I have no idea how to solve or workaround this. The link you added about 13.3 runtime points to 13.3LTS ML. We use 13.3LTS and that one doesn't have the I also don't get the warning (or the error that is solved now) when running locally. I am going to create a Issue in the dbt-databricks github for this. |
|
@thijs-nijhuis-shell sounds good, thanks. If there's some weird interaction on those warnings that turns out to be something we need to fix on the dbt side please do open a separate issue. Hopefully you can get an update that makes those warnings stop. |
Is this a new bug in dbt-core?
Current Behavior
I’m trying to migrate to dbt-core 1.5.1 and play with programmatic invocations
Expected Behavior
can import and use dbtRunner, dbtRunnerResult from python environment
Steps To Reproduce
Relevant log output
Environment
Which database adapter are you using with dbt?
spark
Additional Context
No response
The text was updated successfully, but these errors were encountered: