Bus error since 4.0.4 on a Raspberry Pi

[merdok]

Hi,

the recent update (4.0.4) is causing my nodejs app to crash, i am using https://www.npmjs.com/package/websocket as a dependency to connect to a websocket endpoint, which has this package as dependency.
I was tracking down the issue and found out that the "Bus error" happens during the mask function here https://github.com/theturtle32/WebSocket-Node/blob/master/lib/WebSocketFrame.js#L265

source = <Buffer 7b 22 69 64 22 3a 22 62 61 63 30 33 38 30 36 30 30 30 31 22 2c 22 74 79 70 65 22 3a 22 72 65 71 75 65 73 74 22 2c 22 75 72 69 22 3a 22 73 73 61 70 3a ... 37 more bytes>
mask = <Buffer 48 29 6e 2b>
output = <Buffer 81 d7 48 29 6e 2b 00 00 08 94 21 02 01 00 00 00 88 0f 00 00 2c 00 00 00 88 90 21 02 68 ad 21 02 73 65 74 4e 6f 44 65 6c 61 79 21 02 00 00 00 00 94 3d ... 43 more bytes>
offset = 6
length =87

I have no idea what you changed in version 4.0.4 but when i rollback to version 4.0.3 then it still works.
As far as I understand that some build c program is called to execute those functions faster but apparently it is now broken on some machines. I am running on a Raspberry Pi:
Linux raspberrypi 5.10.52-v7l+ #1441 SMP Tue Aug 3 18:11:56 BST 2021 armv7l GNU/Linux

Can you tell me what the issue here is, and how to fix this? Can it be that the latest update broke this package for raspberry pi devices?

[lpinca]

We rebuilt the binary but that is automated and we made no changes. Maybe the compiler used in https://github.com/dockcross/dockcross/tree/master/linux-armv7 was updated and it produced a different binary.

Can you try to run this script https://github.com/websockets/bufferutil/blob/v4.0.4/package.json#L13, copy the generated binary to your Raspberry Pi and see if it works?

This issue might be similar to #132.

[merdok]

I have tried that and the issue is still there.
But i am pretty certain that something is broken with the binary after your recent update...
To be sure about that the binary has an issue, I renamed the node.napi.armv7.node on my Raspberry Pi to some random name blabla.node, and the "Bus error" is gone and everything works again. I guess when the correct binary cannot be found then it defaults to fallback.js, right?

Maybe something with the docker container changed since you automatically build the binary last time?

[lpinca]

I guess when the correct binary cannot be found then it defaults to fallback.js, right?

Correct.

Maybe something with the docker container changed since you automatically build the binary last time?

Maybe but that is outside of our direct control. What we can do is stop shipping prebuilt binaries for armv6 and armv7 but I don't know if it is a good idea.

[merdok]

I actually did not notice any change in performance with the fallback.js so I am not sure if that is a concern (but I guess that would depend on how heavy it is used)

I mean the prebuilt library was working for arm7 in version 4.0.3. I think you can continue to ship them but you need to actually make sure that they are working especially when you build them automatically, just building and hoping that they will work is quite a bad idea...

The question is now if you can provide a quick fix (by temporary removing the broken binaries) or if you want to find the root cause of the broken binaries and fix the problem there?

[lpinca]

I mean the prebuilt library was working for arm7 in version 4.0.3. I think you can continue to ship them but you need to actually make sure that they are working especially when you build them automatically, just building and hoping that they will work is quite a bad idea...

Testing them on the same machine where they are built will obviously work.

[lpinca]

I think I'll just cut a new patch or minor version removing prebuilt binaries for armv6 and armv7.

[merdok]

That would be fine with me, but you could also revert back to the binaries from 4.0.3

[lpinca]

That would be fine with me, but you could also revert back to the binaries from 4.0.3

Then the next time we cut a new release the binaries will be broken again.

[merdok]

That's why you would need to investigate the root cause of the issue.
Anyway, I am fine with any solution as long as it will work again 😊

[lpinca]

I don't have an armv6 or armv7 device so it's not easy for me to investigate.

[merdok]

The "Bus error" is not really something what tells you what exactly the issue is... if you tell me how I can debug that then I could help you.

[lpinca]

According to https://stackoverflow.com/questions/212466/what-is-a-bus-error-is-it-different-from-a-segmentation-fault, it is a memory alignment issue and if I have to guess, it is caused by dockcross/dockcross@7b63720#diff-f9e3abded6708e2d856adcdaeb77508db9fb89eb724ac8d330cfcac014a5dab0.

[merdok]

I wonder if also other binaries are affected which were then build using the same way. I mean it should not only affect then this package but many more.

[lpinca]

I mean it should not only affect then this package but many more.

Yes, other modules might be affected by the same issue.

Anyway, we are only cross compiling ARM binaries.

[lpinca]

Can I ask you to build the binary on your Raspberry Pi and see if it works?

git clone https://github.com/websockets/bufferutil.git
cd bufferutil
npm install
npm test

[merdok]

Yes, that works! The mocha tests pass and after copying the binary from the build/Release directory to the prebuild/linux-arm directory and renaming it to node.napi.armv7.node no more crashes in nodejs!
So that should confirm that everything is fine with the code, just the build messes something up in the docker container.

> [email protected] test
> mocha



  bindings
    ✔ masks a buffer (1/2)
    ✔ masks a buffer (2/2)
    ✔ unmasks a buffer

  fallback
    ✔ masks a buffer (1/2)
    ✔ masks a buffer (2/2)
    ✔ unmasks a buffer


  6 passing (25ms)

[UncleSamSwiss]

Yes, other modules might be affected by the same issue.

Yes, we are seeing this on Raspberry Pi 2B+, 3B and 4 in at least three different libraries (called adapters) of the ioBroker project that are dependent on either ws or websocket.

[UncleSamSwiss]

BTW: GitHub Actions allows you to run builds directly on ARM. I have never done this, but I would expect that you can (simply) add it to your execution matrix and then gather the binary during the release.

[lpinca]

@UncleSamSwiss there is no support for ARM on hosted runners, see https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners. A self-hosted runner would be needed to do that.

[Apollon77]

It is verifed that deleting the binary fixes the issue. Tries to do a local rebuild or such were not really changeing anything ... so we need a new release of both... thank you for your support here

[lpinca]

A local rebuild should work but you have to replace the prebuilt binary with the new one, otherwise the prebuilt will be used.

[Apollon77]

Ahhh then this was "The trick" ;-)

[lpinca]

I've released [email protected] without prebuilt binaries for Linux ARM. I'm closing this.

[Lyxal12]

Sorry for the question, but i am new to linux/raspberry.

How do i replace the binary?

[lpinca]

@Lyxal12 if you install the latest version (4.0.5) you don't have to replace anything.

[Lyxal12]

I think i installed version 4.0.5 but i still get "instance system.adapter.lgtv.0 terminated due to SIGBUS" warning in my iobroker. Any idea what i can do?

[UncleSamSwiss]

I think i installed version 4.0.5 but i still get "instance system.adapter.lgtv.0 terminated due to SIGBUS" warning in my iobroker. Any idea what i can do?

Please don't ask ioBroker specific questions in this issue, this is a library which is independent of ioBroker. Use the forum or the adapter repository for such questions. Thanks!