available at compile time?

[jjb]

Hello again!

here https://elements.heroku.com/buildpacks/heroku/heroku-buildpack-apt

it says "Add support for apt-based dependencies during both compile and runtime."

But during my compile phase, I see a lot of these:

remote:        ERROR: ld.so: object '/app/.apt/usr/lib/x86_64-linux-gnu/libjemalloc.so.2' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
see full log attached

Is this a bug with the buildpack, or am I not understanding what "compile time" is?

It's not very important in this case, I don't absolutely need jemalloc to be available during compile time. But I would like to understand it, and to not have errors in the logs.

Let me know if you need any more info, thanks!

[colincasey]

@jjb Hard to say if that's a bug or not without more information. For security reasons, buildpacks don't execute with root permissions so the Apt buildpack has to install given packages into non-root locations. For certain packages this method may not work without extra configuration.

To get a better sense of the issue can you provide more details such as:

• Which stack on Heroku are you using?
• What buildpacks are used by your application?
• What packages are listed in your Aptfile?
• How is jemalloc being used?
• Any other log information that might be relevant to share?

[jjb]

Which stack on Heroku are you using?

heroku-22

What buildpacks are used by your application?

1. heroku-community/apt
2. https://github.com/heroku/heroku-buildpack-activestorage-preview
3. heroku/nodejs
4. heroku/ruby

What packages are listed in your Aptfile?

libjemalloc2

How is jemalloc being used?

these two env vars

LD_PRELOAD:                  /app/.apt/usr/lib/x86_64-linux-gnu/libjemalloc.so.2
MALLOC_CONF:                 dirty_decay_ms:1000,narenas:2,background_thread:true

Any other log information that might be relevant to share?

only that it's working at runtime

heroku run bash...
~ $ file /app/.apt/usr/lib/x86_64-linux-gnu/libjemalloc.so.2
/app/.apt/usr/lib/x86_64-linux-gnu/libjemalloc.so.2: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=ffdb577fa78a283f51714728e64d35d30bac41a9, stripped
~ $ MALLOC_CONF="stats_print:true" ruby -e "exit"
___ Begin jemalloc statistics ___
Version: "5.2.1-0-gea6b3e973b477b8061e0076bb257dbd7f3faa756"
...

thanks!

[colincasey]

@jjb I think I see what happening. You've set the environment variable LD_PRELOAD to the path that libjemalloc2 will be installed into.

The compile phase sees the configured LD_PRELOAD path but that file will not exist until the heroku-community/apt buildpack runs and downloads the library or restores it from cache. Until libjemalloc2 is present, anything that calls the dynamic linker (ld.so) will end up trying to use invalid configuration in LD_PRELOAD and emit the error you see.

At runtime, libjemalloc2 will already be present so the configured LD_PRELOAD path will be valid and no error will be emitted.

To eliminate this error during the compile phase you may want to look at creating a .profile script to configure LD_PRELOAD during the runtime phase only.

[jjb]

Hi, thanks for checking. Yes, that is my understanding of the problem. But, I thought that the packages would be installed before my compile happened.

see my original message 😅

here https://elements.heroku.com/buildpacks/heroku/heroku-buildpack-apt
it says "Add support for apt-based dependencies during both compile and runtime."
Is this a bug with the buildpack, or am I not understanding what "compile time" is?

[colincasey]

But, I thought that the packages would be installed before my compile happened.

While it's true that the packages are installed before your application compile step happens, the entire compile phase also includes:

• loading env vars
• executing each of the configured buildpacks in order, which, for your application would be:
  • heroku-community/apt
  • https://github.com/heroku/heroku-buildpack-activestorage-preview
  • heroku/nodejs
  • heroku/ruby

My explanation of the problem was intended to clarify that having LD_PRELOAD set to a non-existent library at the very start of the compile phase will cause errors to be produced by any executable called until heroku-community/apt runs and the library is finally present. Therefore, if you don't need libjemalloc2 during the compile phase then use a .profile script to configure LD_PRELOAD during the runtime phase only.

I'm going to close this issue since this is not a bug. Feel free to reopen if you have more to add @jjb.

[jjb]

thanks so much for your detailed response

cause errors to be produced by any executable called until heroku-community/apt runs and the library is finally present

heroku-community/apt runs first (which is the case in the list you provided above, so maybe i'm missing something)

here's a log up until i hit the error:

https://gist.github.com/jjb/63d485383a7e62289863073369ca0990

theory: do buildpacks not have access to one anothers' state while they are building? so maybe the ruby buildpack doesn't have access to files that the apt buildpack installed?

if this is in some documentation that i scanned over 10 times and now have wasted everyone's time, i'm very sorry 😅

[joshwlewis]

@jjb - Can you try LD_PRELOAD=libjemalloc.so.2 instead of the full path?

[jjb]

Interesting, I didn't realize that was an option.

"If a shared object dependency does not contain a slash...." https://www.man7.org/linux/man-pages/man8/ld.so.8.html#DESCRIPTION

That stopped the error in compile time and did not break runtime 🎉. I'm guess it's becasue

1. the files installed by the apt buildpack are in different paths in compile and runtime
2. LD_LIBRARY_PATH is kept up to date automatically in compile and run time both cases

it also created a new error when opening a run bash session.

➔ heroku run bash -r staging
Running bash on ⬢ bulletin-newsletter-staging... up, run.1537 (Basic)
ERROR: ld.so: object 'libjemalloc.so.2' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
ERROR: ld.so: object 'libjemalloc.so.2' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
~ $ MALLOC_CONF="stats_print:true" ruby -e "exit"
___ Begin jemalloc statistics ___
Version: "5.2.1-0-gea6b3e973b477b8061e0076bb257dbd7f3faa756"
...
~ $ unset LD_PRELOAD
~ $ MALLOC_CONF="stats_print:true" ruby -e "exit"
~ $

Looks like LD_LIBRARY_PATH is set in .profile.d/000_apt.sh - i wonder what ruby thing is running in my shell before that runs. deosn't seem to be coming from other profile init scripts.

~ $ ls .profile.d/
000_activestorage_preview.sh  000_apt.sh  nodejs.sh  ruby.sh  WEB_CONCURRENCY.sh
~ $ cat .profile.d/000_activestorage_preview.sh
export PATH="$HOME/.heroku/activestorage-preview/bin:$PATH"
~ $ unset LD_LIBRARY_PATH
~ $ source .profile.d/*
~ $

I could go back to heroku support with that question, if you think it's not related to the apt buildpack.

Thanks!

[joshwlewis]

That stopped the error in compile time and did not break runtime 🎉. I'm guess it's becasue

• the files installed by the apt buildpack are in different paths in compile and runtime
• LD_LIBRARY_PATH is kept up to date automatically in compile and run time both cases

Yes, that's correct in both cases. You can see $LD_LIBRARY_PATH is set two different ways -- one for later buildpacks, one for runtime:

• https://github.com/heroku/heroku-buildpack-apt/blob/master/bin/compile#L124
• https://github.com/heroku/heroku-buildpack-apt/blob/master/bin/compile#L133

Looks like LD_LIBRARY_PATH is set in .profile.d/000_apt.sh - i wonder what ruby thing is running in my shell before that runs. deosn't seem to be coming from other profile init scripts.

Yes, that's some of our platform tooling that enables heroku run whatever. If the message bothers you, you could take Colin's suggestion above and create your own profile.d script to set LD_PRELOAD at runtime.

[jjb]

gotcha - thanks again to both of you for the info and help here!

i've now updated my gist: https://gist.github.com/jjb/9ff0d3f622c8bbe904fe7a82e35152fc#heroku