Superset fails to run with ModuleNotFoundError: No module named 'cryptography.hazmat.backends.openssl.x509'

[omnipresent07]

After cloning the repository and trying to run in docker on master branch, I get this error:

meltano invoke superset fab create-admin --username admin --firstname lebron --lastname james --email [email protected] --password password
2023-01-03T10:30:34.244420Z [error    ] Traceback (most recent call last):
  File "/workspaces/nba-monte-carlo/.meltano/utilities/superset/venv/bin/superset", line 5, in <module>
    from superset.cli.main import superset
  File "/workspaces/nba-monte-carlo/.meltano/utilities/superset/venv/lib/python3.9/site-packages/superset/__init__.py", line 21, in <module>
    from superset.app import create_app
  File "/workspaces/nba-monte-carlo/.meltano/utilities/superset/venv/lib/python3.9/site-packages/superset/app.py", line 23, in <module>
    from superset.initialization import SupersetAppInitializer
  File "/workspaces/nba-monte-carlo/.meltano/utilities/superset/venv/lib/python3.9/site-packages/superset/initialization/__init__.py", line 33, in <module>
    from superset.extensions import (
  File "/workspaces/nba-monte-carlo/.meltano/utilities/superset/venv/lib/python3.9/site-packages/superset/extensions/__init__.py", line 32, in <module>
    from superset.utils.cache_manager import CacheManager
  File "/workspaces/nba-monte-carlo/.meltano/utilities/superset/venv/lib/python3.9/site-packages/superset/utils/cache_manager.py", line 24, in <module>
    from superset.utils.core import DatasourceType
  File "/workspaces/nba-monte-carlo/.meltano/utilities/superset/venv/lib/python3.9/site-packages/superset/utils/core.py", line 76, in <module>
    from cryptography.hazmat.backends.openssl.x509 import _Certificate
ModuleNotFoundError: No module named 'cryptography.hazmat.backends.openssl.x509'

Need help fixing this problem? Visit http://melta.no/ for troubleshooting steps, or to
join our friendly Slack community.

Superset metadata database could not be initialized: `superset db upgrade` failed
make: *** [Makefile:18: superset-visuals] Error 1
make: *** [docker-run-superset] Error 2

It seems to be similar to error mentioned in #54 but the superset-test branch seems to be gone now. I'm happy to provide a PR if I can be pointed in the right direction on where this might be happening. I am running on macOS Catalina 10.15.7

[matsonj]

What python version?

[omnipresent07]

I'm on 2.7.16 (know i should be on 3) but running all this from docker run so my local python version shouldn't be in conflict ...I think.

[matsonj]

The dockerfile is pinned to python 3.9. Are you running this on the mdsbox image or are you running this on a different image?

[matsonj]

Nvm, I see you are running the makefile. I will take a look at it!

[matsonj]

[matsonj]

ok so the issue is that the docker image is x86, not arm64. So the fix will be to build the docker image on arm64. I don't think I can do this dynamically? @gregwdata can you set docker to build images based on platform of {{ current_machine ))?

[matsonj]

@omnipresent07 as a work around, you can run this in codespaces for now. But I am contemplating how to solve this elegantly (may need a second dockerfile)

[gregwdata]

@matsonj To build an image for ARM64, I think you need to start with a base image that has been built for ARM64. Looking at the base image you used, https://hub.docker.com/r/nikolaik/python-nodejs/tags?page=1 all the OS/ARCH fields say linux/amd64. So there would need to be an alternate source to start from, or go up the chain of dockerfile dependencies and build from scratch.

That said, I don't have experience with cross-platform builds. Did a little searching and found this https://dev.to/lakhansamani/create-docker-image-on-apple-silicon-m1-mac-2f75 which claims you can still build for linux/amd64 and run on M1. Might try docker build command with an explicit --platform linux/amd64 flag and see if that does the trick.

[matsonj]

ah good tip. Ok trying that now - i.e. docker build ... --platform linux/arm64

If this works, will add a command to the make file make docker-build-mac which builds on the arm64 platform. I think this is the only necessary change for this to work!

[matsonj]

ha doesn't appear to be environmental - issue still recurs.

However, I did find this issue, which makes me think that maybe I've messed up the superset version pinning (should be pinned to 1.5.2)

[gregwdata]

If it helps, I had done some updates the the Superset dependencies in #41, meltano.yml, to make it play nice w/ Superset 2.0, since the version pinning was changed from == 1.5.0 to >= 1.5.0 in #38

[matsonj]

ah! yeah ok I'm going to pin to 1.5.2 and try again. lets see what happens here! Could probably handle superset 2.0.x with some pip install modifications in the docker image, but I want to fix this issue first and then move on to handling the version upgrade.

[matsonj]

I was able to reproduce the fix listed in the github issue on the superset side by pinning the dependency cryptography=3.4.7. I've also bumped the version from >=1.5.0 to >=2.0.0.

The issue was not the environment of the docker image, ALTHOUGH it runs incredibly slow on an M1 mac. Just a missing dependency related to superset going GA for a new version.

I also tried pinning superset v1.5.2, but that had another set of dependency issues with Flask / Jinja, so instead it seemed more straight forward to go to superset v2.0.2.

[matsonj]

Its not building inside the github action - so I'm now chasing down a dependency. It appears that it has an old version of numpy that conflicts with superset 2.0.1.

[matsonj]

Ended up cleaning up some other stuff too, but the following line of code was modified, superset is now building again:

https://github.com/matsonj/nba-monte-carlo/pull/65/files#diff-6d0acc7374df2a40ec960634f967ae63128eac507abd089a68b8787e9433ddf2R68