Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WSL: ftruncate does not truncate mmap'd files #902

Closed
freeone3000 opened this issue Aug 15, 2016 · 23 comments
Closed

WSL: ftruncate does not truncate mmap'd files #902

freeone3000 opened this issue Aug 15, 2016 · 23 comments
Labels

Comments

@freeone3000
Copy link

freeone3000 commented Aug 15, 2016

Please use the following bug reporting template to help produce actionable and reproducible issues:

  • A brief description
    ftruncate does not truncate mmap'd files while they are mmap'd.
  • Expected results
    file gets truncated to desired size in bytes, even when mmap'd; SIGBUS occurs if memory outside new file bounds is accessed. (This is the behavior on Ubuntu.)
  • Actual results (with terminal output if applicable)
    file does not get truncated
  • Your Windows build number
    14393
  • Steps / commands required to reproduce the error
#include <sys/mman.h>
#include <fcntl.h>
#include <unistd.h>
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <errno.h>
#define FAIL exit(errno);

int main() {
    /* prepare test data */
    int buf_size = 8102;
    char* buf = malloc(buf_size);
    if(NULL == buf) {
            FAIL
    }
    memset(buf, '?', buf_size);
    int fd = open("test_buffer", O_RDWR | O_CREAT);
    if(fd < 0) {
            FAIL
    }

    /* run test */
    write(fd, buf, buf_size);
    char* mm = mmap(NULL, buf_size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
    if(MAP_FAILED == mm) {
            FAIL
    }
    memset(mm, '!', buf_size);
    int tres = ftruncate(fd, 16);
    if(0 != tres) { /* okay, trying again later */
            write(STDOUT_FILENO, "NOFIRST\n", 9);
            errno = 0;
    }
    int mures = munmap(mm, buf_size);
    if(0 != mures) {
            FAIL
    }
    tres = ftruncate(fd, 16);
    if(0 != tres) {
            write(STDOUT_FILENO, "FAIL BOTH\n", 11);
            exit(1);
    }

    write(STDOUT_FILENO, "SUCCESS\n", 9);
    close(fd);
}
  • Strace of the failing command
  `execve("./min_test.bin", ["./min_test.bin"], [/* 23 vars */]) = 0
  brk(0)                                  = 0x12ab000
  access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
  mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7c97540000
  access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
  open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
  fstat(3, {st_mode=S_IFREG|0644, st_size=64852, ...}) = 0
  mmap(NULL, 64852, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f7c97530000
  close(3)                                = 0
  access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
  open("/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
  read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P \2\0\0\0\0\0"..., 832) = 832
  fstat(3, {st_mode=S_IFREG|0755, st_size=1840928, ...}) = 0
  mmap(NULL, 3949248, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f7c96e30000
  mprotect(0x7f7c96fea000, 2097152, PROT_NONE) = 0
  mmap(0x7f7c971ea000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1ba000) = 0x7f7c971ea000
  mmap(0x7f7c971f0000, 17088, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f7c971f0000
  close(3)                                = 0
  mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7c97520000
  mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7c97510000
  arch_prctl(ARCH_SET_FS, 0x7f7c97510740) = 0
  mprotect(0x7f7c971ea000, 16384, PROT_READ) = 0
  mprotect(0x600000, 4096, PROT_READ)     = 0
  mprotect(0x7f7c97422000, 4096, PROT_READ) = 0
  munmap(0x7f7c97530000, 64852)           = 0
  brk(0)                                  = 0x12ab000
  brk(0x12cd000)                          = 0x12cd000
  open("test_buffer", O_RDWR|O_CREAT, 0112547600) = 3
  write(3, "????????????????????????????????"..., 8102) = 8102
  mmap(NULL, 8102, PROT_READ|PROT_WRITE, MAP_SHARED, 3, 0) = 0x7f7c9754b000
  ftruncate(3, 16)                        = -1 EINVAL (Invalid argument)
  write(1, "NOFIRST\n\0", 9)              = 9
  munmap(0x7f7c9754b000, 8102)            = 0
  ftruncate(3, 16)                        = 0
  write(1, "SUCCESS\n\0", 9)              = 9
  close(3)                                = 0
  exit_group(0)                           = ?
  +++ exited with 0 +++
  • Required packages and commands to install
    apt-get -y install build-essential

See our contributing instructions for assistance.

@stehufntdev
Copy link
Collaborator

Thanks for reporting the issue. This is discussed in #658 so closing it out as a duplicate. Please give us feedback on the user voice page so we can prioritize improving mmap support.

@therealkenc
Copy link
Collaborator

therealkenc commented Mar 3, 2018

De-duping. The NOFIRST path still fires on 17110. Work around appears to be to munmap() the file like this.

gcc mmap-fail.c -o mmap-fail && rm -f test_buffer && strace -f -o mmap-fail.strace ./mmap-fail

WSL:

415   open("test_buffer", O_RDWR|O_CREAT, 0134077600) = 3
415   write(3, "????????????????????????????????"..., 8102) = 8102
415   mmap(NULL, 8102, PROT_READ|PROT_WRITE, MAP_SHARED, 3, 0) = 0x7f48104f4000
415   ftruncate(3, 16)                  = -1 EINVAL (Invalid argument)
415   write(1, "NOFIRST\n\0", 9)        = 9
415   munmap(0x7f48104f4000, 8102)      = 0
415   ftruncate(3, 16)                  = 0
415   write(1, "SUCCESS\n\0", 9)        = 9

Real Linux:

25098 openat(AT_FDCWD, "test_buffer", O_RDWR|O_CREAT, 0171140) = 3
25098 write(3, "????????????????????????????????"..., 8102) = 8102
25098 mmap(NULL, 8102, PROT_READ|PROT_WRITE, MAP_SHARED, 3, 0) = 0x7fe496235000
25098 ftruncate(3, 16)                  = 0
25098 munmap(0x7fe496235000, 8102)      = 0
25098 ftruncate(3, 16)                  = 0
25098 write(1, "SUCCESS\n\0", 9)        = 9

@therealkenc
Copy link
Collaborator

@stehufntdev @benhillis (Apologies for the ping) If this could be looked at on your end it will help put some indeterminate rpm / yum (and in general BerkeleyDB) issues to rest. We're going to get more users using non-Debian derived systems, and BerkeleyDB is used all over Creation. It is a clean repro up top on 17627.

@ytrezq
Copy link

ytrezq commented Mar 30, 2018

I added a full repro. ʀᴘᴍ issues might not be linked to current case…

@Uldiniad
Copy link

Uldiniad commented May 3, 2018

any news on this issue?

@drujd
Copy link

drujd commented May 26, 2018

I too would very much like to see this issue resolved. It blocks building Android/LineageOS on WSL:

[  2% 2392/93534] Generating TOC: /home/ddd/android/lineage/out/target/common/obj/JAVA_LIBRARIES/sdk_v8_intermediates/classes.jar.toc
FAILED: /home/ddd/android/lineage/out/target/common/obj/JAVA_LIBRARIES/sdk_v8_intermediates/classes.jar.toc
/bin/bash -c "(ASAN_OPTIONS=detect_leaks=0 prebuilts/build-tools/linux-x86/bin/ijar /home/ddd/android/lineage/out/target/common/obj/JAVA_LIBRARIES/sdk_v8_intermediates/classes.jar /home/ddd/android/lineage/out/target/common/obj/JAVA_LIBRARIES/sdk_v8_intermediates/classes.jar.toc.tmp ) && (if cmp -s /home/ddd/android/lineage/out/target/common/obj/JAVA_LIBRARIES/sdk_v8_intermediates/classes.jar.toc.tmp /home/ddd/android/lineage/out/target/common/obj/JAVA_LIBRARIES/sdk_v8_intermediates/classes.jar.toc ; then rm /home/ddd/android/lineage/out/target/common/obj/JAVA_LIBRARIES/sdk_v8_intermediates/classes.jar.toc.tmp ; else mv /home/ddd/android/lineage/out/target/common/obj/JAVA_LIBRARIES/sdk_v8_intermediates/classes.jar.toc.tmp /home/ddd/android/lineage/out/target/common/obj/JAVA_LIBRARIES/sdk_v8_intermediates/classes.jar.toc ; fi )"
ftruncate(fd_out, GetSize()): Invalid argument
/bin/bash: line 1: 26107 Aborted                 (core dumped) ( ASAN_OPTIONS=detect_leaks=0 prebuilts/build-tools/linux-x86/bin/ijar /home/ddd/android/lineage/out/target/common/obj/JAVA_LIBRARIES/sdk_v8_intermediates/classes.jar /home/ddd/android/lineage/out/target/common/obj/JAVA_LIBRARIES/sdk_v8_intermediates/classes.jar.toc.tmp )
[  2% 2397/93534] Generating TOC: /home/ddd/android/lineage/out/target/common/obj/JAVA_LIBRARIES/sdk_v17_intermediates/classes.jar.toc
FAILED: /home/ddd/android/lineage/out/target/common/obj/JAVA_LIBRARIES/sdk_v17_intermediates/classes.jar.toc
/bin/bash -c "(ASAN_OPTIONS=detect_leaks=0 prebuilts/build-tools/linux-x86/bin/ijar /home/ddd/android/lineage/out/target/common/obj/JAVA_LIBRARIES/sdk_v17_intermediates/classes.jar /home/ddd/android/lineage/out/target/common/obj/JAVA_LIBRARIES/sdk_v17_intermediates/classes.jar.toc.tmp ) && (if cmp -s /home/ddd/android/lineage/out/target/common/obj/JAVA_LIBRARIES/sdk_v17_intermediates/classes.jar.toc.tmp /home/ddd/android/lineage/out/target/common/obj/JAVA_LIBRARIES/sdk_v17_intermediates/classes.jar.toc ; then rm /home/ddd/android/lineage/out/target/common/obj/JAVA_LIBRARIES/sdk_v17_intermediates/classes.jar.toc.tmp ; else mv /home/ddd/android/lineage/out/target/common/obj/JAVA_LIBRARIES/sdk_v17_intermediates/classes.jar.toc.tmp /home/ddd/android/lineage/out/target/common/obj/JAVA_LIBRARIES/sdk_v17_intermediates/classes.jar.toc ; fi )"
ftruncate(fd_out, GetSize()): Invalid argument
/bin/bash: line 1: 26139 Aborted                 (core dumped) ( ASAN_OPTIONS=detect_leaks=0 prebuilts/build-tools/linux-x86/bin/ijar /home/ddd/android/lineage/out/target/common/obj/JAVA_LIBRARIES/sdk_v17_intermediates/classes.jar /home/ddd/android/lineage/out/target/common/obj/JAVA_LIBRARIES/sdk_v17_intermediates/classes.jar.toc.tmp )
[  2% 2399/93534] Build hyb /home/ddd/android/lineage/out/target/product/gts210velte/obj/ETC/hyph-nb_intermediates/hyph-nb.hyb <- external/hyphenation-patterns/nb//hyph-nb.pat.txt
12585 unique nodes, 42861 total
[  2% 2400/93534] Build hyb /home/ddd/android/lineage/out/target/product/gts210velte/obj/ETC/hyph-nn_intermediates/hyph-nn.hyb <- external/hyphenation-patterns/nn//hyph-nn.pat.txt
12585 unique nodes, 42861 total
[  2% 2401/93534] Build hyb /home/ddd/android/lineage/out/target/product/gts210velte/obj/ETC/hyph-hu_intermediates/hyph-hu.hyb <- external/hyphenation-patterns/hu//hyph-hu.pat.txt
21515 unique nodes, 102669 total
ninja: build stopped: subcommand failed.
12:29:11 ninja failed with: exit status 1

It is also problematic with bup.

@Uldiniad
Copy link

@drujd check my xda thread to bypass the ijar issue: https://forum.xda-developers.com/android/software-hacking/guide-how-to-build-lineageos-15-1-t3750175

@gao4263
Copy link

gao4263 commented Aug 17, 2018

this issue on 16 Aug 2016 , why not fix ?

@sirredbeard
Copy link
Contributor

sirredbeard commented Feb 25, 2019

Noting we have hit this issue in WLinux Enterprise on RHEL and Scientific Linux, specifically to some wonky rpm behavior here and there. See WhitewaterFoundry/Pengwin-Enterprise#20.

@qclzdh
Copy link

qclzdh commented May 7, 2019

I encounter following error when build android with wsl on win10 18890, i also checked on google everywhere, on many page lead me to here, but i can't find any solution here, and also it seem this issue also solved in 18890, is there anyone can help me to solve this>
[ 0% 60/50136] Generating TOC: out/target/common/obj/JAVA_LIBRARIES/sdk_v8_intermediates/classes.jar.toc FAILED: out/target/common/obj/JAVA_LIBRARIES/sdk_v8_intermediates/classes.jar.toc /bin/bash -c "(ASAN_OPTIONS=detect_leaks=0 prebuilts/build-tools/linux-x86/bin/ijar out/target/common/obj/JAVA_LIBRARIES/sdk_v8_intermediates/classes.jar out/target/common/obj/JAVA_LIBRARIES/sdk_v8_intermediates/classes.jar.toc.tmp ) && (if cmp -s out/target/common/obj/JAVA_LIBRARIES/sdk_v8_intermediates/classes.jar.toc.tmp out/target/common/obj/JAVA_LIBRARIES/sdk_v8_intermediates/classes.jar.toc ; then rm out/target/common/obj/JAVA_LIBRARIES/sdk_v8_intermediates/classes.jar.toc.tmp ; else mv out/target/common/obj/JAVA_LIBRARIES/sdk_v8_intermediates/classes.jar.toc.tmp out/target/common/obj/JAVA_LIBRARIES/sdk_v8_intermediates/classes.jar.toc ; fi )" ftruncate(fd_out, GetSize()): Invalid argument /bin/bash: line 1: 1072 Aborted (core dumped) ( ASAN_OPTIONS=detect_leaks=0 prebuilts/build-tools/linux-x86/bin/ijar out/target/common/obj/JAVA_LIBRARIES/sdk_v8_intermediates/classes.jar out/target/common/obj/JAVA_LIBRARIES/sdk_v8_intermediates/classes.jar.toc.tmp ) [ 0% 61/50136] Install: out/host/linux-x86/bin/aapt2

@thorsteneb
Copy link

thorsteneb commented May 7, 2019

If you haven’t seen it: https://www.howtogeek.com/413564/windows-10-is-getting-a-built-in-linux-kernel/

[deleted bits about WSL1 no longer being developed alongside wsl2 - I jumped the gun there]

I’m curious how the BerkeleyDB stuff is going to work since the Windows kernel doesn’t allow the kind of file system shenanigans that Linux does, and WSL files are available from both Windows and Linux. Maybe they’ll be kept in a lightweight VM now and just accessed from Windows.

My understanding is this kernel is the same or a very similar approach to what Azure does, so here’s to a fully compatible Linux experience on Windows without needing to spin up a full fat VM.

@ytrezq
Copy link

ytrezq commented May 7, 2019

@yorickdowne : I still think this issue needs to be fixed. We’re not talking about a feature but the fact the implementation isn’t implementing something correctly.

There’s 3 points which are unclear with the new plan : will it remain possible to access Windows files like currently or cygwin ? Will it be required to manage a disk image it’s own file system with different sizes than the host drive (currently there’s no need for large images which mean free space of wsl or cygwin can be efficiently shared without having too much free space on image leaving no space on the host and vice versa). Will it be possible to still see each Linux process from Windows like with pico process currently ?

As currently phrased, this really sounds like leaving all those possibilities and switching to something with the same limitations as https://en.wikipedia.org/wiki/Cooperative_Linux (something far heavier than a Linux Docker). In that sense the plan to keep wsl1 really sounds to be a red flag for me indicating that huge benefits of the current implementation will be lost meaning Cygwin will remains superior in functionality over both implementations. Which in turns means this bug should be fixed (not being even able to run rpm as a single command but seriously).

@drujd
Copy link

drujd commented May 7, 2019

@yorickdowne Nooooo! :(

So is it as I feared - WSL1 will stop being supported/developed?!

WSL2 is a glorified Linux VM. :( And, unless I'm wrong, it will require Hyper-V, therefore blocking (due to Microsoft's Hyper-V implementation) other virtualization providers, along with other problems stemming from running 'host' in a VM (albeit in a root partition).

@drujd
Copy link

drujd commented May 7, 2019

@ytrezq Agreed, judging from the information available, WSL2 is a cop out compared to WSL1.

@therealkenc
Copy link
Collaborator

WSL1 will stop being supported/developed?!

The announcement goes out of its way to say: "Individual Linux distros can be run either as a WSL 1 distro, or as a WSL 2 distro, can be upgraded or downgraded at any time, and you can run WSL 1 and WSL 2 distros side by side."

Saying WSL1 will stop being supported, immediately anyway, is pretty overstated. For example:

I’m curious how the BerkeleyDB stuff is going to work

Actually it isn't clear whether BDB is still broken, since #3939 was squashed. Squashed just a little over a month ago. See 'cause the devs are still fixing issues in WSL1. Fixing issues at the same time the devs were frantically getting WSL2 in a state a screencap animated gif could be taken for Build 2019.

[BDB might still be broken in some subtle unreported way, mind. Dunno. Haven't looked. No well formed issue submission to look at, even for the curiosity value.]

Meanwhile this issue (#902) is still unresolved, of course. Says so at the top of the screen and everything.

There’s 3 points which are unclear with the new plan

There are lots of points about the new plan, both of the mostly clear and of the somewhat unclear variety. None of which have to do with this open issue specifically. The notable question we're faced with today is whether the WSL2 announcement implodes every favorite open issue into full chat mode for the next couple of months.

@qclzdh
Copy link

qclzdh commented Jun 19, 2019

This issue was solved on wsl2, i can build android successfully now.

I encounter following error when build android with wsl on win10 18890, i also checked on google everywhere, on many page lead me to here, but i can't find any solution here, and also it seem this issue also solved in 18890, is there anyone can help me to solve this>
[ 0% 60/50136] Generating TOC: out/target/common/obj/JAVA_LIBRARIES/sdk_v8_intermediates/classes.jar.toc FAILED: out/target/common/obj/JAVA_LIBRARIES/sdk_v8_intermediates/classes.jar.toc /bin/bash -c "(ASAN_OPTIONS=detect_leaks=0 prebuilts/build-tools/linux-x86/bin/ijar out/target/common/obj/JAVA_LIBRARIES/sdk_v8_intermediates/classes.jar out/target/common/obj/JAVA_LIBRARIES/sdk_v8_intermediates/classes.jar.toc.tmp ) && (if cmp -s out/target/common/obj/JAVA_LIBRARIES/sdk_v8_intermediates/classes.jar.toc.tmp out/target/common/obj/JAVA_LIBRARIES/sdk_v8_intermediates/classes.jar.toc ; then rm out/target/common/obj/JAVA_LIBRARIES/sdk_v8_intermediates/classes.jar.toc.tmp ; else mv out/target/common/obj/JAVA_LIBRARIES/sdk_v8_intermediates/classes.jar.toc.tmp out/target/common/obj/JAVA_LIBRARIES/sdk_v8_intermediates/classes.jar.toc ; fi )" ftruncate(fd_out, GetSize()): Invalid argument /bin/bash: line 1: 1072 Aborted (core dumped) ( ASAN_OPTIONS=detect_leaks=0 prebuilts/build-tools/linux-x86/bin/ijar out/target/common/obj/JAVA_LIBRARIES/sdk_v8_intermediates/classes.jar out/target/common/obj/JAVA_LIBRARIES/sdk_v8_intermediates/classes.jar.toc.tmp ) [ 0% 61/50136] Install: out/host/linux-x86/bin/aapt2

@ytrezq
Copy link

ytrezq commented Jul 20, 2019

Agreed, judging from the information available, WSL2 is a cop out compared to WSL1.

@drujd Yikes ! It requires a file based ᴠʜᴅ Like any Windows Virtual Machine (the huge slow because of fileystem on hard drives because of the double layer of fragmentation and likely to make file system access slower than ᴡꜱʟ1). I would also add that I’ve more than 256Gb of code in my ᴡꜱʟ1 home directory simply barring access to ᴡꜱʟ2.
It’s definitely just a modern Microsoft Version of https://en.wikipedia.org/wiki/Cooperative_Linux

I also have many apps through otvdm which uses the Intel Haxm ᴀᴘɪ which is very unlikely to support Hyper-V.

Unless the ability to boot real Linux main rootfs from ᴡꜱʟ2 is given (boot from ʙᴛʀꜰꜱ or Linux ʟᴠᴍ or regular partition), this issue should definitely be fixed.

@gordonliang
Copy link

I got the same issue while compiling webrtc on WSL 1, I fix this issue with a workaround solution: munmap the mmap'd file before ftruncate and it's works fine for me.

@rubin55
Copy link

rubin55 commented Dec 27, 2022

@therealkenc will this issue ever be fixed? I think a lot of users have specific + legitimate requirements to use WSL1 instead of WSL2; this issue has a lot of impact on various sofwares (re: the many tickets closed as duplicates to this one). Is this issue particularly hard to fix or won't MS give it priority? It makes WSL1 borderline unusable imho.. hoping hard for a fix.

@Daniel-Khodabakhsh
Copy link

@drujd check my xda thread to bypass the ijar issue: https://forum.xda-developers.com/android/software-hacking/guide-how-to-build-lineageos-15-1-t3750175

The link now shows a post that has been updated to only include WSL2 steps. But Waybackmachine is here to help: https://web.archive.org/web/20180617170133/https://forum.xda-developers.com/android/software-hacking/guide-how-to-build-lineageos-15-1-t3750175 however there's still some digging required.

Basically you need to apply this change to the synced ijar/zip.cc file: https://review.lineageos.org/c/LineageOS/android_build/+/208102. Make sure to make clean before re-attempting the brunch command.

Copy link
Contributor

This issue has been automatically closed since it has not had any activity for the past year. If you're still experiencing this issue please re-file this as a new issue or feature request.

Thank you!

1 similar comment
Copy link
Contributor

This issue has been automatically closed since it has not had any activity for the past year. If you're still experiencing this issue please re-file this as a new issue or feature request.

Thank you!

@drujd
Copy link

drujd commented Mar 14, 2024

So I guess that's it, the final confirmation (not that we needed one), WSL1 really is dead in the water, replaced by a glorified VM... :(

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests