Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Out of memory on wayland platform #538

Closed
tonyakiki opened this issue Feb 17, 2023 · 9 comments
Closed

Out of memory on wayland platform #538

tonyakiki opened this issue Feb 17, 2023 · 9 comments
Labels
bug Something isn't working upstream Needs support from an upstream

Comments

@tonyakiki
Copy link

tonyakiki commented Feb 17, 2023 •
edited
Loading

We have noticed a memory leak when running cog with WL platform. It is a Yocto Linux with wayland/weston and Cog/wpewebkit showing a full screen website with some video playing together with some other graphics and the HW is an intel based platform.

We can see with /proc/meminfo that the memory is listed as “Cached” and “Unevictable”. With /sys/kernel/debug/dri/0/i915_gem_objects we can see that shrinkable objects and size goes slowly up and closely match the values in meminfo. It constantly goes up and down a bit but over time it goes up with 1-2 objects every 10 sec or so. The machine with 4GB reaches OOM after 1-2h.

Running versions:

  • cog 0.16.1 (WPE WebKit 2.38.3)
  • vainfo: VA-API version: 1.14 (libva 2.14.0)
  • vainfo: Driver version: Intel iHD driver for Intel(R) Gen Graphics - 22.3.1 (6547f4658)

Some observations we have seen so far.

  • We don’t see the problem on all Intel platforms for instance it works on Intel Broxton (Gen9) but we do have problem on 
    • Intel Geminilake (Gen9)
    • Intel Elkhartlake (Gen11)
    • Intel Jasperlake (Gen11)
  • We see the problem on some web sites but other not, but they can look very similar with video etc.
  • We don’t see the problem if we turn off Weston and running Cog/webkit with drm platform.
  • The memory/gem objects are released if Cog is closed.

Typically a while before the OOM notice the high buff/cache
image
@philn philn added bug Something isn't working needs investigation Needs some investigation to find the cause of the issue labels Feb 17, 2023
@philn
Copy link
Member

philn commented Feb 17, 2023

The fact that the leak happens on some platforms but not all you tested might indicate a driver issue. Needs further investigation...

@tonyakiki
Copy link
Author

As I mentioned previously the memory leak does not exist on DRM platform. How can we investigate further if this is a driver issue or cog / webkit issue?
I run cog with WEBKIT_SAMPLE_MEMORY=1 and i could confirm the leak

@aperezdc
Copy link
Member

aperezdc commented Feb 28, 2023 •
edited
Loading

@tonyakiki Which versions/commits of WPEBackend-fdo and Cog were you using when the leak was observed? We have recently landed Igalia/WPEBackend-fdo#178 and Igalia/WPEBackend-fdo#181 around the time of your original report—I wonder if you have found a regression, or if you are using a packaged release maybe the first of these PRs solves the issue.

@tonyakiki
Copy link
Author

tonyakiki commented Mar 1, 2023 •
edited
Loading

I am using:
cog 0.16.1 (d2f817c)
wpebackend-fdo 1.14.0 (966b37d)
I'll try these patches and get back with a report. Thanks for your advice!

@aperezdc
Copy link
Member

aperezdc commented Mar 1, 2023

I am using:
cog 0.16.1 (d2f817c)
wpebackend-fdo 1.14.0 (966b37d)

These are indeed from before the fixes.

I'll try these patches and get back with a report. Thanks for your advice!

Excellent, thanks! Looking forward to your results, and hoping that the issue is gone 👍🏼

@tonyakiki
Copy link
Author

I confirm that Igalia/WPEBackend-fdo#178 and Igalia/WPEBackend-fdo#181 fix the memory leak problem.
@aperezdc when can we expect a release of these fixes?

@aperezdc aperezdc added upstream Needs support from an upstream and removed needs investigation Needs some investigation to find the cause of the issue labels Mar 4, 2023
@aperezdc
Copy link
Member

aperezdc commented Mar 4, 2023

I confirm that Igalia/WPEBackend-fdo#178 and Igalia/WPEBackend-fdo#181 fix the memory leak problem.
@aperezdc when can we expect a release of these fixes?

Thanks a lot for confirming that these patches fixed the issue! I have just tagged and uploaded WPEBackend-fdo 1.14.1, which includes them.

@aperezdc aperezdc closed this as completed Mar 4, 2023
@tonyakiki
Copy link
Author

Hello again @aperezdc ! I am using meta-webkit and I see that /recipes-browser/wpebackend-fdo still missing the last release 1.14.1. Can you confirm that? and when will it be released there?

@aperezdc
Copy link
Member

aperezdc commented Mar 7, 2023

Hello again @aperezdc ! I am using meta-webkit and I see that /recipes-browser/wpebackend-fdo still missing the last release 1.14.1. Can you confirm that? and when will it be released there?

I have just submitted a PR to get the update into meta-webkit: Igalia/meta-webkit#455 — it should be quick to have it reviewed and merged, hopefully.

clopez added a commit to clopez/WPEBackend-fdo that referenced this issue Jun 25, 2024
@clopez
REGRESSION(b51f539): Fix memory leak when the buffer is destroyed at …
1307fbf 
…view-backend-exportable-fdo-egl

Since commit b51f539 there is a memory leak each time the wl_resource is destroyed.
This can be easily reproduced by repeteadly switching full-screen on/off
(pressing F11 key) with Cog on Weston.

The memory leak is caused because since b51f539 the wpe_fdo_egl_exported_image object
is not cleaned anymore on the bufferDestroyListenerCallback callback.

Commit cb6b86a fixed the leak but introduced crashes on some cases, so it was reverted.

This is a new attempt at fixing this leak, this adds safeguards to ensure that the
image object is not cleaned twice or with the wrong exported status.

Related-to: Igalia#73 Igalia#175 Igalia#176 Igalia#178
Related-to: Igalia/cog#538
@clopez clopez mentioned this issue Jun 25, 2024
Merged
clopez added a commit to Igalia/WPEBackend-fdo that referenced this issue Sep 11, 2024
@clopez
REGRESSION(b51f539): Fix memory leak when the buffer is destroyed at …
5b1c5e4 
…view-backend-exportable-fdo-egl

Since commit b51f539 there is a memory leak each time the wl_resource is destroyed.
This can be easily reproduced by repeteadly switching full-screen on/off
(pressing F11 key) with Cog on Weston.

The memory leak is caused because since b51f539 the wpe_fdo_egl_exported_image object
is not cleaned anymore on the bufferDestroyListenerCallback callback.

Commit cb6b86a fixed the leak but introduced crashes on some cases, so it was reverted.

This is a new attempt at fixing this leak, this adds safeguards to ensure that the
image object is not cleaned twice or with the wrong exported status.

Related-to: #73 #175 #176 #178
Related-to: Igalia/cog#538
aperezdc pushed a commit to Igalia/WPEBackend-fdo that referenced this issue Sep 11, 2024
@clopez @aperezdc
REGRESSION(b51f539): Fix memory leak when the buffer is destroyed at …
af922a8 
…view-backend-exportable-fdo-egl

Since commit b51f539 there is a memory leak each time the wl_resource is destroyed.
This can be easily reproduced by repeteadly switching full-screen on/off
(pressing F11 key) with Cog on Weston.

The memory leak is caused because since b51f539 the wpe_fdo_egl_exported_image object
is not cleaned anymore on the bufferDestroyListenerCallback callback.

Commit cb6b86a fixed the leak but introduced crashes on some cases, so it was reverted.

This is a new attempt at fixing this leak, this adds safeguards to ensure that the
image object is not cleaned twice or with the wrong exported status.

Related-to: #73 #175 #176 #178
Related-to: Igalia/cog#538
(cherry picked from commit 5b1c5e4)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working upstream Needs support from an upstream
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@philn @aperezdc @tonyakiki