Support SQLAlchemy 2.0 #84

phillipuniverse · 2024-01-06T20:56:10Z

Fixes #83.

I am going to be running these in production soon and I figured I would go ahead and contribute back.

My process was to take the fork from @bodik who did most of the heavy lifting (thanks @bodik!) and then applied fixes to other SQLAlchemy 2.0 issues. My 1 change so far fixes this exception:

                    except (InvalidRequestError, AttributeError):
                    # query might not contain columns yet, hence cannot be compiled
                    # or query might be a sqla2.0 select statement
                    pass
                # also try to infer the models from various internals
    >           for table_tuple in query._setup_joins + query._legacy_setup_joins:
    E           AttributeError: 'Select' object has no attribute '_legacy_setup_joins'

Not exactly sure where to go from here, I haven't yet investigated the testing ecosystem of this library but probably we want to get SQLAlchemy 2.0 into the test suite.

@bodik @tomviner what do you think?

phillipuniverse · 2024-01-06T21:10:34Z

Looking at the other open PRs, this also contains the fix to #67 by way of #63 (which @bodik had already applied to their fork).

Under some circunstances, such as prepping an aliased subquery to be joined later on, the model entity is missing, provoking an error when trying to apply filters. (cherry picked from commit edc7c24)

…tyle?) (cherry picked from commit b6791e3)

(cherry picked from commit 203d401)

phillipuniverse · 2024-01-09T15:31:03Z

@tomviner FYI I rebased on the latest master to get the MariaDB pin to pass the GH workflows

tomviner · 2024-01-14T17:15:59Z

@tomviner FYI I rebased on the latest master to get the MariaDB pin to pass the GH workflows

Thanks for your PR, and rebase. The tests for SQLA 1.4 are failing. Would you like to take a look at fixing them?

phillipuniverse · 2024-01-14T18:29:46Z

@tomviner no problem I think I see the issue. Glad we’ve got good tests covering it!

I can also take a look at how to add SQLAlchemy 2 to CI

This property is removed in SQLAlchemy 2.x

… .gitignore This allows Tox to find all Python versions when using pyenv

drop_all() makes the 'bind' positional argument required

…s drivers

Specify compatible versions of SQLAlchemy and SQLAlchemy-Utils for Python, SQLAlchemy versions

phillipuniverse

@tomviner sorry it took me a bit but I have a fix up!

I also included a lot of changes to Tox and how tests run in CI. Probably the most impactful change is that with all the modifications I made to tox.ini, we can remove the Tox environment compatibility from the GitHub workflow config. The result is that we now have SQLAlchemy 2 running in CI and an easy to way to run it locally!

There are still a few test failures around loads and joins in SQLAlchemy 2. It will likely take me another week to get some bandwidth to fix these additional tests. Hopefully there's a straightforward fix!

phillipuniverse · 2024-01-21T04:39:40Z

.github/workflows/tests.yml

-          # sqlalchemylatest (i.e. > 2.0.0) is not yet supported
-          # for any version of python
-
-          - {python: '3.7', tox: "py37-sqlalchemy1.0"}
-          - {python: '3.7', tox: "py37-sqlalchemy1.1"}
-          - {python: '3.7', tox: "py37-sqlalchemy1.2"}
-          - {python: '3.7', tox: "py37-sqlalchemy1.3"}
-          - {python: '3.7', tox: "py37-sqlalchemy1.4"}
-
-          - {python: '3.8', tox: "py38-sqlalchemy1.0"}
-          - {python: '3.8', tox: "py38-sqlalchemy1.1"}
-          - {python: '3.8', tox: "py38-sqlalchemy1.2"}
-          - {python: '3.8', tox: "py38-sqlalchemy1.3"}
-          - {python: '3.8', tox: "py38-sqlalchemy1.4"}
-
-          - {python: '3.9', tox: "py39-sqlalchemy1.0"}
-          - {python: '3.9', tox: "py39-sqlalchemy1.1"}
-          - {python: '3.9', tox: "py39-sqlalchemy1.2"}
-          - {python: '3.9', tox: "py39-sqlalchemy1.3"}
-          - {python: '3.9', tox: "py39-sqlalchemy1.4"}
-
-          # python3.10 with sqlalchemy <= 1.1 errors with:
-          # AttributeError: module 'collections' has no attribute 'MutableMapping'
-          - {python: '3.10', tox: "py310-sqlalchemy1.2"}
-          - {python: '3.10', tox: "py310-sqlalchemy1.3"}
-          - {python: '3.10', tox: "py310-sqlalchemy1.4"}


No longer necessary since this is now fully encoded in tox.ini.

phillipuniverse · 2024-01-21T04:40:50Z

.python-version

+3.7.17
+3.8.18
+3.9.18
+3.10.13
+3.11.7
+3.12.1


I locally use pyenv and thought this would be useful to specify. Happy to remove if this muddies the waters for tooling.

phillipuniverse · 2024-01-21T04:41:36Z

setup.py

-        'mysql': ['mysql-connector-python-rf==2.2.2'],
-        'postgresql': ['psycopg2==2.8.4'],
+        'mysql': ['mysql-connector-python-rf>=2.2.2'],
+        'postgresql': ['psycopg2>=2.8.4'],


I don't exactly recall the Python version that was building wheels. I didn't think the versions were all that important for validating proper execution. I can look into this deeper if necessary.

phillipuniverse · 2024-01-21T04:44:53Z

sqlalchemy_filters/models.py

-    return orm_descriptor.extension_type == symbol('HYBRID_PROPERTY')
+    # SQLAlchemy 2 treats extension_type as an enum, not a symbol(). Enum is at sqlalchemy.ext.hybrid.HybridExtensionType
+
+    return str(orm_descriptor.extension_type) in ("symbol('HYBRID_PROPERTY')", 'HybridExtensionType.HYBRID_PROPERTY')


This is a little bit hacky but I was trying to optimize for performance. Other things I thought of:

Use isinstance() to see if orm_descriptor.extension_type is an Enum and then use .name to compare. isinstance() can be really slow in a hot path, and this gets invoked a lot for all properties on queried models

Try/except import since HybridExtensionType only exists in 2+. Kind of obtuse, I thought the same "hot path" reason to exclude this one too

Happy to change for a different opinion on the right way to check this.

phillipuniverse · 2024-01-21T04:45:16Z

test/conftest.py

@@ -131,7 +131,7 @@ def connection(db_uri, db_engine_options, is_postgresql):

    yield connection

-    Base.metadata.drop_all()
+    Base.metadata.drop_all(engine)


SA 2.x compatibility

phillipuniverse · 2024-01-21T04:48:46Z

tox.ini

@@ -10,11 +19,14 @@ extras =
    mysql
    postgresql
 deps =
-    {py37,py38,py39,py310}: sqlalchemy-utils~=0.37.8
+    !sqlalchemy2,!sqlalchemylatest: sqlalchemy-utils~=0.37.8


I really wanted !sqlalchemy{2,latest} to work but it didn't. Alas.

Simplifies tox expressions and doesn't change anything fundamental about this library. 0.38.3 dropped support for Python 3.4 and 3.5 which sqlalchemy-filters never supported in the first place

trbtm · 2024-02-06T12:25:17Z

Is there an ETA on this?

trbtm · 2024-04-12T13:39:18Z

@tomviner is there any work open that I can support with in order to release this?

ConfusedCoder556 · 2024-07-03T15:58:45Z

Any updates?

Ar0010r · 2024-09-19T13:17:22Z

I there anything I could help with in order to merge this ?

This was referenced Jan 6, 2024

Support SQLAlchemy 2.X #83

Open

Compatibility with select() syntax #79

Open

javimudi and others added 3 commits January 9, 2024 09:30

fix(models): Check for None entity when getting query models

220d092

Under some circunstances, such as prepping an aliased subquery to be joined later on, the model entity is missing, provoking an error when trying to apply filters. (cherry picked from commit edc7c24)

sqlalchemy 1.4 infer tables also from select object (sqlalchemy 2.0 s…

66e3941

…tyle?) (cherry picked from commit b6791e3)

sqlalchemy 1.4 infer tables also from select object (fix auto_join)

7e0854a

(cherry picked from commit 203d401)

phillipuniverse force-pushed the sqlalchemy-2.x branch from b5de041 to 984d5d3 Compare January 9, 2024 15:30

phillipuniverse force-pushed the sqlalchemy-2.x branch from 984d5d3 to 976b5a9 Compare January 20, 2024 16:58

phillipuniverse added 7 commits January 20, 2024 22:22

Conditional access of _legacy_setup_joins property

c94c862

This property is removed in SQLAlchemy 2.x

Gitignore venv and local dev tool directories

db58232

Tox with SQLAlchemy 2

407f7c3

Add all compatible Python versions to .python-version and remove from…

dc7786e

… .gitignore This allows Tox to find all Python versions when using pyenv

Move tox to default dev dependencies

b45a30c

SQLAlchemy 2 compatibility with drop_all

7fcb4c1

drop_all() makes the 'bind' positional argument required

Avoid building wheels on later Python versions with mysql and postgre…

a951949

…s drivers

phillipuniverse force-pushed the sqlalchemy-2.x branch from 516ef7d to 8328dd8 Compare January 21, 2024 04:22

phillipuniverse added 3 commits January 20, 2024 22:47

Configure Tox for SQLAlchemy 2 and Python 3.11, 3.12

389b17c

Specify compatible versions of SQLAlchemy and SQLAlchemy-Utils for Python, SQLAlchemy versions

SQLAlchemy 2 compatibility for hybrid properties and methods

debf911

Move all Tox env configuration and version compatibility into tox.ini

a0dea62

phillipuniverse force-pushed the sqlalchemy-2.x branch from fe6188f to a0dea62 Compare January 21, 2024 04:47

phillipuniverse commented Jan 21, 2024

View reviewed changes

phillipuniverse added 2 commits January 20, 2024 23:01

Fix tox expression for sqlalchemy-utils dep

d7d2246

SQLAlchemy 2 swapped order of columns in SELECT statement

526375f

phillipuniverse force-pushed the sqlalchemy-2.x branch from 4e5a3a1 to 526375f Compare January 21, 2024 05:59

phillipuniverse added 3 commits January 22, 2024 09:33

Use tox version from setup.py

39f54fc

Use Tox 3-compatible version of tox-gh-actions

bcd4848

Fix flake8

73bed5e

phillipuniverse added 2 commits January 22, 2024 09:48

Add Python 3.11 and 3.12 to CI

9fd5f99

Minimum SQLAlchemy-Utils 0.38.3

1865f0d

Simplifies tox expressions and doesn't change anything fundamental about this library. 0.38.3 dropped support for Python 3.4 and 3.5 which sqlalchemy-filters never supported in the first place

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support SQLAlchemy 2.0 #84

Support SQLAlchemy 2.0 #84

phillipuniverse commented Jan 6, 2024

phillipuniverse commented Jan 6, 2024

phillipuniverse commented Jan 9, 2024

tomviner commented Jan 14, 2024

phillipuniverse commented Jan 14, 2024

phillipuniverse left a comment

phillipuniverse Jan 21, 2024

phillipuniverse Jan 21, 2024

phillipuniverse Jan 21, 2024

phillipuniverse Jan 21, 2024

phillipuniverse Jan 21, 2024

phillipuniverse Jan 21, 2024

trbtm commented Feb 6, 2024

trbtm commented Apr 12, 2024

ConfusedCoder556 commented Jul 3, 2024

Ar0010r commented Sep 19, 2024

+.7.17
+.8.18
+.9.18
+.10.13
+.11.7
+.12.1

Support SQLAlchemy 2.0 #84

Are you sure you want to change the base?

Support SQLAlchemy 2.0 #84

Conversation

phillipuniverse commented Jan 6, 2024

phillipuniverse commented Jan 6, 2024

phillipuniverse commented Jan 9, 2024

tomviner commented Jan 14, 2024

phillipuniverse commented Jan 14, 2024

phillipuniverse left a comment

Choose a reason for hiding this comment

phillipuniverse Jan 21, 2024

Choose a reason for hiding this comment

phillipuniverse Jan 21, 2024

Choose a reason for hiding this comment

phillipuniverse Jan 21, 2024

Choose a reason for hiding this comment

phillipuniverse Jan 21, 2024

Choose a reason for hiding this comment

phillipuniverse Jan 21, 2024

Choose a reason for hiding this comment

phillipuniverse Jan 21, 2024

Choose a reason for hiding this comment

trbtm commented Feb 6, 2024

trbtm commented Apr 12, 2024

ConfusedCoder556 commented Jul 3, 2024

Ar0010r commented Sep 19, 2024