Skip to content
This repository has been archived by the owner on Jul 13, 2023. It is now read-only.

Track down memory change issue in autoendpoint in 1.28.0 #917

Closed
bbangert opened this issue Jun 13, 2017 · 5 comments · Fixed by #942
Closed

Track down memory change issue in autoendpoint in 1.28.0 #917

bbangert opened this issue Jun 13, 2017 · 5 comments · Fixed by #942
Assignees
@pjenvey
Labels
bug Operational p2 ready
Milestone
E2.3 Maintain/Imp...

Comments

@bbangert
Copy link
Member

Starting with 1.28.0 release, autoendpoint's memory usage has skyrocketed. We need to track down and fix the memory issue as it causes issues with the log shipper when the memory is severely impacted.
@pjenvey pjenvey self-assigned this Jun 15, 2017
@pjenvey
Copy link
Member

pjenvey commented Jun 20, 2017

Also #802

The likeliest culprit that changed in 1.28.0 was #854, which switched our VAPID verification from pycrypto to cryptography

@alex
Copy link
Contributor

alex commented Jun 20, 2017

FWIW, in the meantime you can probably add some semi-targetted __pypy__.add_memory_pressure yourself to improve the situation. Anywhere in VerifyJWT.decode with a value of 128 will probably have a noticable impact.

pjenvey added a commit that referenced this issue Jun 28, 2017
@pjenvey
fix: add memory pressure for jwt's cryptography usage
bd6e32f 
related to cffi issue #320: pypy's GC isn't aware of many of
cryptography's allocations, manually add pressure for now so it keeps
up

Closes #917
pjenvey added a commit that referenced this issue Jun 28, 2017
@pjenvey
fix: add memory pressure for jwt's cryptography usage
ebc4695 
related to cffi issue #320: pypy's GC isn't aware of many of
cryptography's allocations, manually add pressure for now so it keeps
up

Closes #917
@pjenvey pjenvey mentioned this issue Jun 28, 2017
Closed
pjenvey added a commit that referenced this issue Jun 28, 2017
@pjenvey
fix: add memory pressure for jwt's cryptography usage
6719a32 
related to cffi issue #320: pypy's GC isn't aware of many of
cryptography's allocations, manually add pressure for now so it keeps
up

Closes #917
pjenvey added a commit that referenced this issue Jun 28, 2017
@pjenvey
fix: add memory pressure for jwt's cryptography usage
4eb82ec 
related to cffi issue #320: pypy's GC isn't aware of many of
cryptography's allocations, manually add pressure for now so it keeps
up

temporarily toggleable so we can experiment w/ it vs the cryptography
recent leak fix (cryptography PR 3732)

Closes #917
pjenvey added a commit that referenced this issue Jun 28, 2017
@pjenvey
fix: add memory pressure for jwt's cryptography usage
99048cd 
related to cffi issue #320: pypy's GC isn't aware of many of
cryptography's allocations, manually add pressure for now so it keeps
up

temporarily toggleable so we can experiment w/ it vs the cryptography
recent leak fix (cryptography PR 3732)

Closes #917
@pjenvey pjenvey mentioned this issue Jun 28, 2017
Merged
pjenvey added a commit that referenced this issue Jul 26, 2017
@pjenvey
chore: upgrade to latest cryptography
e0ae4cb 
Issue #917
@pjenvey pjenvey mentioned this issue Jul 26, 2017
Merged
@pjenvey
Copy link
Member

pjenvey commented Aug 2, 2017

this isn't quite tracked down yet

@pjenvey pjenvey reopened this Aug 2, 2017
@pjenvey pjenvey mentioned this issue Aug 2, 2017
Closed
3 tasks
@pjenvey pjenvey added the ready label Aug 2, 2017
@jrconlin jrconlin added p2 and removed p1 labels Aug 16, 2017
@pjenvey pjenvey mentioned this issue Oct 6, 2017
Closed
@pjenvey pjenvey mentioned this issue Apr 24, 2018
Closed
@bbangert
Copy link
Member Author

What are the specific criteria for closing this?

@pjenvey
Copy link
Member

pjenvey commented May 31, 2018

See #802: Our slow leaks are due to likely fragmentation: pypy's memory allocation on top of glibc malloc can cause this slow leak. A MALLOC_ARENA_MAX=2 has helped the endpoint nodes a lot, further experimentation w/ jemalloc is needed, tracked in: mozilla-services/push-service#79 and https://bugzilla.mozilla.org/show_bug.cgi?id=1442705

@pjenvey pjenvey closed this as completed May 31, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@pjenvey pjenvey

Labels
bug Operational p2 ready
Projects
None yet
Milestone
E2.3 Maintain/Improve endpoint messag...
Development

Successfully merging a pull request may close this issue.

fix: add memory pressure for jwt's cryptography usage mozilla-services/autopush
4 participants
@alex @pjenvey @jrconlin @bbangert