Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove mmapSize check, it has wrong value on ARM64 #281

Merged
merged 1 commit into from
Sep 1, 2021

Conversation

yarysh
Copy link

@yarysh yarysh commented Aug 20, 2021

In this statement:
mmapAddr, mmapSize, errno := unix.Syscall6(unix.SYS_MREMAP,...)

mmapSize - it is not an allocated size, at least for ARM64 arch.
For ARM64, mmapSize will be equal to the old size.

Example script: https://gist.github.com/yarysh/2b4db19ebe1236f31ff6780d956628f5

I run this script on AMD64 machine with Ubuntu 20.04, and it returns:

Requested size: 20
mmapAddr: 824634507264
mmapSize: 20

And on ARM64 machine (Raspberry PI and AWS Graviton) with Ubuntu 20.04, it returns:

Requested size: 20
mmapAddr: 274878390272
mmapSize: 1024
mremap size mismatch: requested: 20 got: 1024

I have checked man for mremap and it describes returns as:

RETURN VALUE
       On  success mremap() returns a pointer to the new virtual memory area.  On error,
       the value MAP_FAILED (that is, (void *) -1) is returned, and errno is set appropriately.

This issue also affected badger on ARM64: https://discuss.dgraph.io/t/error-mremap-size-mismatch-on-arm64/15333


This change is Reviewable

@CLAassistant
Copy link

CLAassistant commented Aug 20, 2021

CLA assistant check
All committers have signed the CLA.

Copy link
Contributor

@manishrjain manishrjain left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

Reviewed 1 of 1 files at r1, all commit messages.
Reviewable status: :shipit: complete! all files reviewed, all discussions resolved (waiting on @yarysh)

@manishrjain manishrjain merged commit c2901dc into dgraph-io:master Sep 1, 2021
46bit added a commit to 46bit/distributed-systems that referenced this pull request Sep 18, 2021
I hope this will fix the following critical error that I have seen
whenever writing a lot of data to Badger on my Raspberry Pi:

  Unable to truncate file: "[…]/000001.vlog" error: mremap size mismatch: requested: 271382146 got: 536870912

This error causes data loss (yikes.) It seems to have been fixed in
Ristretto already [1] so I'm going to try that.

[1] dgraph-io/ristretto#281
46bit added a commit to 46bit/distributed-systems that referenced this pull request Sep 18, 2021
I hope this will fix the following critical error that I have seen
whenever writing a lot of data to Badger on my Raspberry Pi:

  Unable to truncate file: "[…]/000001.vlog" error: mremap size mismatch: requested: 271382146 got: 536870912

This error causes data loss (yikes.) It seems to have been fixed in
Ristretto already [1] so I'm going to try that.

[1] dgraph-io/ristretto#281
@awfm9
Copy link

awfm9 commented Feb 22, 2022

@manishrjain could you please release this fix? It has been merged almost half a year ago, and there is still no release with a fix.

@manishrjain
Copy link
Contributor

Hi. I’m no longer maintaining these repos, as I’m no longer with Dgraph Labs. We have fork of this repo that we maintain at outcaste-io. Feel free to file an issue there and we can make a release.

lithp pushed a commit to lithp/ristretto that referenced this pull request Jun 21, 2022
joshua-goldstein pushed a commit that referenced this pull request Sep 18, 2022
joshua-goldstein pushed a commit that referenced this pull request Oct 10, 2022
This PR adds a specific arm64 implementation for the `mremap` function
which ignores the second return argument from the syscall. **All other
Linux implementations are unchanged.**

This PR reworks [an earlier
PR](#281) so that the
functionality on non-arm64 platforms remains the same.

Tested on Linux amd64 and Linux arm64.
mangalaman93 pushed a commit that referenced this pull request Dec 11, 2023
In #281 mmap size check has been disabled for the ARM64 arch. 

But we at [anytype](https://github.com/anyproto) quite often face the
same problem on the arm arch on Android devices (armeabi-v7a target)
with the [latest badger
v4.2.0](https://github.com/dgraph-io/badger/releases/tag/v4.2.0) and
[latest ristretto v0.1.1](https://github.com/dgraph-io/ristretto).

```memtables error: while opening fid: 1 error: while updating skiplist error: mremap size mismatch: requested: 1479692 got: 67108864```

This PR extends the #281 to cover all arm architectures.

Here is one more mention of this issue
dgraph-io/badger#2027 (comment)

It would also make sense to create the PR on top of v0.1.1 release, but there is not release branch to base on
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

4 participants