Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Memory usage increases per page load #270

Open
ITwrx opened this issue Oct 26, 2020 · 10 comments
Open

Memory usage increases per page load #270

ITwrx opened this issue Oct 26, 2020 · 10 comments

Comments

@ITwrx
Copy link

ITwrx commented Oct 26, 2020

I'm porting my own site/app to Jester and after getting a substantial amount done, i noticed it was using ~800MB of ram. Investigating, i see that every request+response (page load) causes more ram to be used. At first, i noticed this due to the fact that i'm serving videos via html5 video on some pages, and the ram usage goes up quickly, evidently due to #181. Prior to this, i was using httpbeast, but it couldn't serve the videos at all due to #241, so i switched to asynchttpserver.
Now, to try and rule out my code, i'm just testing with a simple route

routes:
  get "/":
    resp INDEX_HTML.format("")

and serving an html string

const INDEX_HTML = """
  <html>                                                                                                                                                      
  <head>
  <meta name="title" content="ITwrx-Jester" />
  <link href="http://ITwrx-Jester/assets/css/ITwrx.css" rel="stylesheet">
  </head>
  <body>
  <h1>Responding with html from separate file with Jester.</h1>                                                                                                <body>                                                                                                                                                      
  <p>This is only a test. If this were a real paragraph, it would be much more interesting.</p>                                                               
  </body>                                                                                                                                                     
  </html>
  """

and it still uses more ram per page load, just much less than a video page.

Both httpbeast and asynchttpserver exhibit this behaviour, but it's more aggressive with asynchttpserver. I've tried nim 1.0.10, 1.2.8 and 1.4.0 and it happens with each. I'm using Jester 0.5.0 on fedora 32. I think it was happening with 0.4.3 too, IIRC. I'm currently running with:
nim c -d:ssl -d:useStdlib -r ITwrxJester.nim
ssl is needed for smtp in another route.

I tried debugging with the built in profiler and it successfully saved a text file once, but i can't interpret it. Also, that profiler report was from when i was surfing the whole site, not just the index page, if it matters. If you want a profile from just the simplified test, let me know.

https://pastebin.com/P3QX8FQc

I also got a report from valgrind, but i have no experience with that and it makes very little sense to me. Let me know if you want me to attach that, assuming it won't have sensitive info in it (i'm not sure what all is in a typical callgrind.out).

I'm still hoping i'm doing something wrong.

Thanks
@dom96
Copy link
Owner

dom96 commented Oct 26, 2020

Have you tried different GCs? Maybe --gc:markAndSweep or --gc:orc?

For a simple HTML this definitely shouldn't happen (how much increase do you see?). For larger files I can see that being real as this kind of thing hasn't been tested much and is likely a result of how async streams work in Nim.

@ITwrx
Copy link
Author

ITwrx commented Oct 26, 2020

I've tried whatever the default for 1.0.10 - 1.40 is and --gc:orc. I will try --gc:markAndSweep and see if that climbs too. With the default and --gc:orc It starts out at around 2.5MB and climbs to ~800MB which is usually when i stop. How fast it climbs seems to be dependent on how much data is in the pages and their assets. It climbs by ~0.3MB for the simple html page IIRC. Pages with videos seem to climb based on how fat the video is and it's much faster (10X?).

@ITwrx
Copy link
Author

ITwrx commented Oct 26, 2020

It still happens with the simple html page and --gc:markAndSweep. The html page has a background image of ~28k and the currently uncompressed css is 31.8k, just for completeness.

@ITwrx
Copy link
Author

ITwrx commented Oct 27, 2020

FYI, i just commented out all the imports except jester and strutils, all my views except the testing index page, and it still occurs.

@Araq
Copy link
Contributor

Araq commented Oct 27, 2020

More things you can try:

  • Compile with --gc:boehm.
  • Compile with --gc:orc -d:useMalloc.
  • Report it on Nim's github issue tracker, preferably the version with as few dependencies as possible.

@ITwrx
Copy link
Author

ITwrx commented Oct 27, 2020

@Araq thanks, i tried it with both gc flags you suggested after removing the link to the css file which included the background image. i also removed -d:ssl which was being used for smtp, just to rule that out. It still climbs, but with only html in the page, it takes a little longer. One may have to perform 5-10 page loads to see it climb by 0.1-0.2MB. The more data in the page being served, the faster it climbs. I will report in the nim tracker.

@ITwrx ITwrx mentioned this issue Oct 27, 2020
Closed
@ITwrx
Copy link
Author

ITwrx commented Oct 28, 2020

@dom96 this seems to be a jester issue, as i made a simple asynchttpserver-only test serving the same html, and it didn't use more ram per page load. Details in the linked issue above.

@dom96
Copy link
Owner

dom96 commented Oct 28, 2020

@ITwrx in that case I'd appreciate it if you could grab the jester source and cut down the fat until you find the ram doesn't increase anymore, that way you can find the culprit in Jester :)

@ITwrx
Copy link
Author

ITwrx commented Oct 28, 2020

i would like to help, but i'm new to nim and i don't know how qualified i am to help determine what the issue is between asynchttpserver, httpbeast and jester is, at this time. IOW, i'll probably take a look, but i wouldn't hold my breath for a breakthrough. :)

@ITwrx
Copy link
Author

ITwrx commented Oct 28, 2020

i just tested the "hello world" example from the jester readme and the mem usage climbs with that too.

This was referenced Oct 30, 2020
Open
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@dom96 @Araq @ITwrx