Upgrade Python dependencies #593

mabuelhagag · 2022-06-29T02:08:06Z

Changelog

Python version upgraded in Dockerfiles
docker/app/Dockerfile
docker/ci/Dockerfile
docker/client/Dockerfile
docker/nginx/Dockerfile
Upgraded requirements
requirements.txt
requirements-webapp.txt
requirements-dev.txt
Flask:
- Moved management CLI commands from manage.py that uses click to actions.py that uses Flask's CLI custom commands as flask-scripts is now deprecated:
  .dockerignore
  MANIFEST.in
  docker/client/Dockerfile
  docker/client/run-lokole.sh
  install.py
  opwen_email_client/webapp/commands.py
  setup.py
- Moved create_app() to __init__ to fix cyclical import error
  opwen_email_client/webapp/init.py
  opwen_email_client/webapp/ioc.py
  tests/opwen_email_client/webapp/base.py
CI/CD
- Disabled CD of the server.
  .github/workflows/cd.yml
- Disable some integration tests as they were failing on live testing (to be reviewed later)
  docker/integtest/tests.sh
- Used docker compose command instead of docker-compose command as it fails on newer versions of Docker
  makefile
- Build Azure Service bus credentials without URL-encode them as it's handled in Kombu. encoding them caused the live test to fail when Celery was trying to connect
  opwen_email_server/config.py
  tests/opwen_email_server/test_config.py
- Running register_client celery task asynchronously caused the tests to report false success results. So now it's called synchronously (to be further investigated later)
  opwen_email_server/integration/connexion.py
- Migrated to ServiceBusAdministrationClient in delete_queues cli command. See migration guide.
  opwen_email_server/integration/cli.py
- Other changes done in Restructure workflows #597
Babel and Jijnja
- Removed the usage of autoescape and with_ extensions as they're now already included in the compiler
  babel.cfg
Connexion
- tests/opwen_email_server/services/test_auth.py
Yapf code formating

holta · 2022-06-29T12:28:57Z

Do you understand the 3 CI test problems above? (Pasted in here...)

jvonau · 2022-06-29T13:51:39Z

@mabuelhagag This is looking a pile better with the updates applied, does make build complete when run on your branch?
@holta This is where I mentioned using a git checkout in place of using pypy as the source for IIAB installs.
7104375646 from the makefile would be related to

.github.env: .github.env.gpg
	@gpg --decrypt --batch --passphrase "$(GPG_PASSPHRASE)" .github.env.gpg >.github.env

github-env: .github.env
	@docker run --rm python:3.7 python -c 'import uuid; print("SUFFIX=%s" % uuid.uuid4())' >>"$(GITHUB_ENV)"
	@sed 's/^export //' <.github.env >>"$(GITHUB_ENV)"

not sure how/where $(GPG_PASSPHRASE) comes from, hope that helps.

… included in the compiler pallets/jinja#1203

codecov-commenter · 2022-06-29T23:51:58Z

Codecov Report

Merging #593 (4890699) into master (3bf8167) will increase coverage by 1.98%.
The diff coverage is 63.96%.

@@            Coverage Diff             @@
##           master     #593      +/-   ##
==========================================
+ Coverage   76.71%   78.69%   +1.98%     
==========================================
  Files          45       52       +7     
  Lines        2766     3394     +628     
==========================================
+ Hits         2122     2671     +549     
- Misses        644      723      +79

Impacted Files	Coverage Δ
opwen_email_client/domain/email/attachment.py	`0.00% <ø> (ø)`
opwen_email_client/domain/email/client.py	`54.34% <ø> (ø)`
opwen_email_client/domain/email/sql_store.py	`100.00% <ø> (ø)`
opwen_email_client/domain/email/store.py	`97.18% <ø> (ø)`
opwen_email_client/domain/email/sync.py	`90.90% <ø> (-0.07%)`	⬇️
opwen_email_client/domain/email/user_store.py	`97.05% <ø> (ø)`
opwen_email_client/util/serialization.py	`100.00% <ø> (ø)`
opwen_email_client/util/wtforms.py	`93.75% <ø> (ø)`
opwen_email_client/webapp/forms/settings.py	`42.10% <ø> (+0.77%)`	⬆️
opwen_email_client/webapp/ioc.py	`71.79% <ø> (-6.64%)`	⬇️
... and 40 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

mabuelhagag · 2022-06-29T23:56:42Z

@mabuelhagag This is looking a pile better with the updates applied, does make build complete when run on your branch? @holta This is where I mentioned using a git checkout in place of using pypy as the source for IIAB installs. 7104375646 from the makefile would be related to
.github.env: .github.env.gpg
	@gpg --decrypt --batch --passphrase "$(GPG_PASSPHRASE)" .github.env.gpg >.github.env

github-env: .github.env
	@docker run --rm python:3.7 python -c 'import uuid; print("SUFFIX=%s" % uuid.uuid4())' >>"$(GITHUB_ENV)"
	@sed 's/^export //' <.github.env >>"$(GITHUB_ENV)"
not sure how/where $(GPG_PASSPHRASE) comes from, hope that helps.

I think this is a CI variable to decrypt the .github.env.gpg for live test. @c-w Can you provide us with more information on this?

jvonau · 2022-06-30T00:19:43Z

Nice, making progress

holta · 2022-08-04T21:35:06Z

@mabuelhagag do you want your code reviewed quickly by @c-w or @sbathgate or anybody else?

(If so, what are the key changes / key files that QA volunteers should be looking at?)

This reverts commit 0515338.

This reverts commit 3b535df.

mabuelhagag · 2022-08-04T21:47:06Z

@c-w @sbathgate

mabuelhagag · 2022-08-05T00:29:58Z

Updated the PR body with changelog

tests/opwen_email_server/utils/test_unique.py

mabuelhagag · 2022-08-05T15:58:20Z

@c-w code formatting is reverted now.

holta · 2022-08-05T16:06:17Z

I just noticed that @mabuelhagag posted a CHANGELOG for this PR at the very top of the discussion here — to help everyone understand:

Upgrade Python dependencies #593 (comment)

Thank you Mohamad for your 4 months of dedication, with this very intensive work of yours, getting to know a non-simple community product over a third of a year.

Most people would have given up a long time ago.
This is a testament to your character.

Such people are hard to find in today's TikTok-driven world of 10-second videos.
I guess Egypt built the pyramids[*] before any other country for a reason!

[*] Though I have to tell you, Mexico's pyramid is larger than any of Egypt's. By volume anyway (-:

holta · 2022-08-05T19:08:03Z

@c-w code formatting is reverted now.

Thanks @mabuelhagag!

And feel free to lint &/or stylize your code for readability in a later / separate PR.

😺

c-w

I don’t have a horse in this race anymore so feel free to disregard this review. Obviously a lot of work went into this, but I can’t skip the feeling that it’s too much in one go. The integration tests are essentially the only sanity check for this project so disabling them is dangerous. I’d rather see an effort to keep them green on each change and perhaps smaller more incremental updates.

c-w · 2022-08-05T20:56:54Z

docker/integtest/tests.sh

 "${scriptdir}/3-receive-email-for-client.sh" && wait_seconds "${TEST_STEP_DELAY}"
-"${scriptdir}/4-client-downloads-emails.sh"


It’s dangerous to delete these tests for a change that should be a no-op (upgrading versions). Maybe there’s too many changes in one go bundled here and there’s an actual semantic change that sneaked in?

I strongly agree with you and I intend to investigate those errors at the earliest chance possible.
However, as we're focused on making the client work now, I will be skipping the server deployment at this PR.

At a later stage when I work on the server, we should review the failures and get them back on.

mabuelhagag · 2022-08-06T15:11:55Z

I don’t have a horse in this race anymore so feel free to disregard this review. Obviously a lot of work went into this, but I can’t skip the feeling that it’s too much in one go. The integration tests are essentially the only sanity check for this project so disabling them is dangerous. I’d rather see an effort to keep them green on each change and perhaps smaller more incremental updates.

@c-w That's exactly what I intend to do moving forward. Changes should be more atomic.

This reverts commit f3359ea.

holta · 2022-08-09T04:20:36Z

Congrats on Lokole 0.8.1 !
https://github.com/ascoderu/lokole/releases/tag/0.8.1

Any MVP ("Minimal Viable Postage") setup suggestions for volunteers interested in trying/testing — e.g. with https://Internet-in-a-Box.org ?

sbathgate · 2022-08-10T15:46:03Z

Great work here, apologies for the absence, been fairly overwhelmed the last few months. I'll try to keep a more diligent eye on future PR's as they arise here. Thanks for the immense effort though.

mabuelhagag added 16 commits June 23, 2022 22:20

upgrade requirements.txt

02b0c9e

upgrade requirements-dev.txt

cf73e48

upgrade requirements-webapp.txt

6b29f9a

refactored pyzmail36 requirement to pass setup-tools build

fae97f6

refactored code to fix dependencies upgrading errors

01057cf

added fasteners requirement for apache-libcloud

d16f673

upgrade SQLAlchemy

ddc7747

moved Flask app initialization to __init__.py to fix circular imports

4cdf77f

copy manage.py code to flask CLI

0642b1c

temporarily un-ignore built static files

3b535df

commit built static files

0515338

upgraded Flask-Caching

974549d

added upgrade TODO

74b0632

changed createadmin command parameters to required options

1c9163d

removed manage.py

af5d0d5

removed unused import

acd1389

mabuelhagag added 9 commits June 29, 2022 17:57

isort files

6efcb0d

refactored import in test_auth.py

d453a15

applied yapf

08556ff

upgraded requests dep

ca10d72

installed requests stubs

76090aa

fixed createadmin command call

7bb10b8

removed autoescape and with_ extensions in Jinja2 as they are already…

c434f99

… included in the compiler pallets/jinja#1203

install git for pip requirements using git in Dockerfile

e00b619

pin git version

d78cbcd

mabuelhagag added 2 commits August 4, 2022 23:39

Revert "commit built static files"

3994344

This reverts commit 0515338.

Revert "temporarily un-ignore built static files"

e28fd53

This reverts commit 3b535df.

mabuelhagag added safe to test and removed safe to test labels Aug 4, 2022

mabuelhagag added 4 commits August 5, 2022 00:27

upgraded requirements-dev.txt

e89817b

upgraded requiremets-webapp.txt

6b587b1

upgraded requirements.txt

587f911

removed typing package as it's a backport for Python <3.5

7638c4c

mabuelhagag added safe to test and removed safe to test labels Aug 4, 2022

mabuelhagag added 2 commits August 5, 2022 01:12

removed redundant conditional in ci.yml

d221539

removed installation of git as pyzmail36 package is now published

5d80264

mabuelhagag added safe to test and removed safe to test labels Aug 4, 2022

c-w reviewed Aug 5, 2022

View reviewed changes

tests/opwen_email_server/utils/test_unique.py Show resolved Hide resolved

unapplied yapf

f3359ea

c-w reviewed Aug 5, 2022

View reviewed changes

Revert "unapplied yapf"

4890699

This reverts commit f3359ea.

mabuelhagag added safe to test and removed safe to test labels Aug 9, 2022

mabuelhagag merged commit c0dd2a9 into ascoderu:master Aug 9, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade Python dependencies #593

Upgrade Python dependencies #593

mabuelhagag commented Jun 29, 2022 •

edited

Loading

holta commented Jun 29, 2022

jvonau commented Jun 29, 2022

codecov-commenter commented Jun 29, 2022 •

edited

Loading

mabuelhagag commented Jun 29, 2022

jvonau commented Jun 30, 2022

holta commented Aug 4, 2022 •

edited

Loading

mabuelhagag commented Aug 4, 2022

mabuelhagag commented Aug 5, 2022

mabuelhagag commented Aug 5, 2022

holta commented Aug 5, 2022 •

edited

Loading

holta commented Aug 5, 2022

c-w left a comment

c-w Aug 5, 2022

mabuelhagag Aug 6, 2022

mabuelhagag commented Aug 6, 2022

holta commented Aug 9, 2022

sbathgate commented Aug 10, 2022

		"${scriptdir}/3-receive-email-for-client.sh" && wait_seconds "${TEST_STEP_DELAY}"
		"${scriptdir}/4-client-downloads-emails.sh"

Upgrade Python dependencies #593

Upgrade Python dependencies #593

Conversation

mabuelhagag commented Jun 29, 2022 • edited Loading

Changelog

holta commented Jun 29, 2022

jvonau commented Jun 29, 2022

codecov-commenter commented Jun 29, 2022 • edited Loading

Codecov Report

mabuelhagag commented Jun 29, 2022

jvonau commented Jun 30, 2022

holta commented Aug 4, 2022 • edited Loading

mabuelhagag commented Aug 4, 2022

mabuelhagag commented Aug 5, 2022

mabuelhagag commented Aug 5, 2022

holta commented Aug 5, 2022 • edited Loading

holta commented Aug 5, 2022

c-w left a comment

Choose a reason for hiding this comment

c-w Aug 5, 2022

Choose a reason for hiding this comment

mabuelhagag Aug 6, 2022

Choose a reason for hiding this comment

mabuelhagag commented Aug 6, 2022

holta commented Aug 9, 2022

sbathgate commented Aug 10, 2022

mabuelhagag commented Jun 29, 2022 •

edited

Loading

codecov-commenter commented Jun 29, 2022 •

edited

Loading

holta commented Aug 4, 2022 •

edited

Loading

holta commented Aug 5, 2022 •

edited

Loading