open() on ref to many MBs long scalar, then <$fh>, causes fatal OOM in pp_readline and sv_gets, probably Win32-only

[bulk88]

open() on a large scalar (MBs worth), with LF lines, NOT CRLF lines, then calling <$fh> aka pp_readline(), causes a mem leak and a fatal "Out of memory!" on 5.32. 5.41 error is similar but more verbose. Tested both 5.41.5 32b and strawberry-perl 5.32.1.1 32b. Both fatal OOM.

Call stack at realloc() fail return NULL and with 5.41.5 blead perl. Line numbers are fuzzy since this is a -O1/release perl build.

	perl541.dll!VMem::Realloc(void * pMem, unsigned int size) Line 199	C++
 	[Inline Frame] perl541.dll!CPerlHost::Realloc(void *) Line 60	C++
 	perl541.dll!PerlMemRealloc(IPerlMem * piPerl, void * ptr, unsigned int size) Line 303	C++
 	perl541.dll!Perl_safesysrealloc(void * where, unsigned int size) Line 307	C
 	perl541.dll!Perl_sv_grow(interpreter * my_perl, sv * const sv, unsigned int newlen) Line 1425	C
 	perl541.dll!Perl_sv_gets(interpreter * my_perl, sv * const sv, _PerlIO * * const fp, int append) Line 9116	C
 	perl541.dll!Perl_do_readline(interpreter * my_perl) Line 4215	C
 	perl541.dll!Perl_runops_standard(interpreter * my_perl) Line 41	C
 	perl541.dll!S_run_body(interpreter * my_perl, long oldscope) Line 2873	C
 	perl541.dll!perl_run(interpreter * my_perl) Line 2779	C
 	perl541.dll!RunPerl(int argc, char * * argv, char * * env) Line 202	C++
 	[Inline Frame] perl.exe!invoke_main() Line 78	C++
 	perl.exe!__scrt_common_main_seh() Line 288	C++
 	kernel32.dll!@BaseThreadInitThunk@12�()	Unknown
 	ntdll.dll!___RtlUserThreadStart@8�()	Unknown
 	ntdll.dll!__RtlUserThreadStart@8�()	Unknown

The toxic sv_grow() is called from

        if (shortbuffered) {		/* oh well, must extend */
            /* we didn't have enough room to fit the line into the target buffer
             * so we must extend the target buffer and keep going */
            cnt = shortbuffered;
            shortbuffered = 0;
            bpx = bp - (STDCHAR*)SvPVX_const(sv); /* box up before relocation */
            SvCUR_set(sv, bpx);
            /* extned the target sv's buffer so it can hold the full read-ahead buffer */
            SvGROW(sv, SvLEN(sv) + append + cnt + 2);
            bp = (STDCHAR*)SvPVX_const(sv) + bpx; /* unbox after relocation */
            continue;
        }

The 14MB realloc count I THINK comes from

    /* get the number of bytes remaining in the read-ahead buffer
     * on first call on a given fp this will return 0.*/
    cnt = PerlIO_get_cnt(fp);

since the open() on $scalar, the PerlIO backend is just returning the whole scalar's len. I'm not sure that readline() having an infinity buffer. is the best design, since in my case, for a private biz app, I was doing open() on $scalar, and $scalar was a RO mmap string from File::Map. The bug does not involve File::Map. My repro script doesn't use File::Map. The bug is more specifically the loops in readline() or sv_gets() .

        if (gimme == G_LIST) {
            if (SvLEN(sv) - SvCUR(sv) > 20) {
                SvPV_shrink_to_cur(sv);
            }
            /* XXX on RC builds, push on stack rather than mortalize ? */
            sv = sv_2mortal(newSV(80));
            continue;
        }

I suspect the SvPV_shrink_to_cur(sv); isn't really working on Win32 or there is a perl level leak until next FREETMPS/NEXTSTATE, in the 2 loops (in pp_readline() and sv_gets()). Note in the attached screenshots, the 14 MB alloc over and over. Im not able to fully debug this, but either the Perl SVPV shrink is broken, and the block isn't being shrunk by malloc()/HeapAlloc/RtlHeapWhatever, and stays at the full 14MB for the rest of the lifetime of the malloc block. Or the 2 loops are actually leaking.

Note Win32 realloc/HeapReAlloc MIGHT BE totally incapable of shrinking buffers (an "optimization"), or HeapReAlloc adds the "14MB-80??? bytes" "released" "free-ed 4096 byte memory pages" to the <= 16KB/64KB/128KB/512KB user mode alloc pools ( https://www.blackhat.com/docs/us-16/materials/us-16-Yason-Windows-10-Segment-Heap-Internals-wp.pdf ). But Perl keeps asking for "14MB" chunks, and Win32 Heap by design/policy/API/ABI (oh no on ABI) back compat, all 14MB allocs must be raw rounded up 4096 pages VM and must come from the kernel raw VM page allocator. So maybe "fragmentation" or the 512KB rule causes OOM on 32b perl. I didn't test the repro on 64b perl.

To repeat, IDK if this is a Perl level leak until FREETMPS or a Win32 Heap API problem, I have 3 hypothesis tho described above. In any case its an OOM and a pretty bad bug, since it is rooted in CRLF/LF and pretty simple to accidentally trigger, but someone could argue that open() on a scalar is very rare and perf degrad/poor design, since you might as well use index() and substr() for perf if the whole file is already in a scalar string.

I did observe the realloc() len arg, was slowly dropping 1 or 100 ish or 500 ish bytes per loop iteration. Screenshot shows the VM OS alloc eventually starts dropping in 4KB pages which confirms the pp_readline() or sv_gets() leak is O(n^2) ish but slowly drops.

Im leaving this crash to someone else, there are too many optimizations and tricks in pp_readline and sv_gets, and I didn't spot an obv quick fix to the leak. Plus my quick glance left more design questions, then answers, because of all the lvalue PV buffer swapping/save stacking/IDK what optimizations in there. I thought/hypo-ed that changing pp_readline() to a 80 byte or 256 or 1024ish byte buffer limit might superficially stop the OOM, but still be leaving leaks until NEXTSTATE/FREETMPS in the code, so I dont see a instant simple fix. Plus design debate, about malloc/memcpying, the ENTIRE!!! PerlIO open( )on $scalar's backend SVPV, from rvalue to lvalue, Even if src backend SVPV is 100's of MBs long!!!! hence this tkt.

Steps to Reproduce

Note the 32b perl, the 16MB input file len , and the LF input file, and binmode :raw. Since I didnt test this on Linux, this might be a Win32 only Perl bug and not repro on Linux, since Perl default record sep must be CRLF, and my input is LF. I didn't test the crash script with a 16MB CRLF input .csv. The "14MB" input I mention, was the original private CSV file that lead to this bug. The 16MB .csv in the repro script same style input.

use File::Slurp;
if(!-e "bigcsv.csv") {
  my @lines;
  for(0..0xFFFF) {
    my $line = '';


    do {
      $line .= (int(rand(0x8000000))+"");
    } while (length $line < 256);
    push(@lines, substr($line,0, 255));
  }
  write_file("bigcsv.csv", { binmode => ':raw' }, join("\x0a", @lines));
  print "made bigcsv run this script again";
  exit 0;
}
my $fh;
my $bin = read_file("bigcsv.csv", { binmode => ':raw' });
die "open" if !open($fh, '<', \$bin);
foreach(<$fh>){
  print $_;
}

5.32 fatal OOM output

C:\sources\plrl>perl crash.pl
Out of memory!

C:\sources\plrl>

Expected behavior

<$fh> returns separated string lines, or returns undef. Not fatal OOM.

Perl configuration

Also OOM fails on 32b 5.41.5.

C:\Users\Owner>perl -V
Summary of my perl5 (revision 5 version 32 subversion 1) configuration:

  Platform:
    osname=MSWin32
    osvers=10.0.19042.746
    archname=MSWin32-x86-multi-thread-64int
    uname='Win32 strawberry-perl 5.32.1.1 #1 Sun Jan 24 12:17:47 2021 i386'
    config_args='undef'
    hint=recommended
    useposix=true
    d_sigaction=undef
    useithreads=define
    usemultiplicity=define
    use64bitint=define
    use64bitall=undef
    uselongdouble=undef
    usemymalloc=n
    default_inc_excludes_dot=define
    bincompat5005=undef
  Compiler:
    cc='gcc'
    ccflags =' -DWIN32 -D__USE_MINGW_ANSI_STDIO -DPERL_TEXTMODE_SCRIPTS -DPERL_I
MPLICIT_CONTEXT -DPERL_IMPLICIT_SYS -DUSE_PERLIO -fwrapv -fno-strict-aliasing -m
ms-bitfields'
    optimize='-s -O2'
    cppflags='-DWIN32'
    ccversion=''
    gccversion='8.3.0'
    gccosandvers=''
    intsize=4
    longsize=4
    ptrsize=4
    doublesize=8
    byteorder=12345678
    doublekind=3
    d_longlong=define
    longlongsize=8
    d_longdbl=define
    longdblsize=12
    longdblkind=3
    ivtype='long long'
    ivsize=8
    nvtype='double'
    nvsize=8
    Off_t='long long'
    lseeksize=8
    alignbytes=8
    prototype=define
  Linker and Libraries:
    ld='g++'
    ldflags ='-s -L"C:\STRAWB~1\perl\lib\CORE" -L"C:\STRAWB~1\c\lib"'
    libpth=C:\STRAWB~1\c\lib C:\STRAWB~1\c\i686-w64-mingw32\lib C:\STRAWB~1\c\li
b\gcc\i686-w64-mingw32\8.3.0
    libs= -lmoldname -lkernel32 -luser32 -lgdi32 -lwinspool -lcomdlg32 -ladvapi3
2 -lshell32 -lole32 -loleaut32 -lnetapi32 -luuid -lws2_32 -lmpr -lwinmm -lversio
n -lodbc32 -lodbccp32 -lcomctl32
    perllibs= -lmoldname -lkernel32 -luser32 -lgdi32 -lwinspool -lcomdlg32 -ladv
api32 -lshell32 -lole32 -loleaut32 -lnetapi32 -luuid -lws2_32 -lmpr -lwinmm -lve
rsion -lodbc32 -lodbccp32 -lcomctl32
    libc=
    so=dll
    useshrplib=true
    libperl=libperl532.a
    gnulibc_version=''
  Dynamic Linking:
    dlsrc=dl_win32.xs
    dlext=xs.dll
    d_dlsymun=undef
    ccdlflags=' '
    cccdlflags=' '
    lddlflags='-mdll -s -L"C:\STRAWB~1\perl\lib\CORE" -L"C:\STRAWB~1\c\lib"'


Characteristics of this binary (from libperl):
  Compile-time options:
    HAS_TIMES
    HAVE_INTERP_INTERN
    MULTIPLICITY
    PERLIO_LAYERS
    PERL_COPY_ON_WRITE
    PERL_DONT_CREATE_GVSV
    PERL_IMPLICIT_CONTEXT
    PERL_IMPLICIT_SYS
    PERL_MALLOC_WRAP
    PERL_OP_PARENT
    PERL_PRESERVE_IVUV
    USE_64_BIT_INT
    USE_ITHREADS
    USE_LARGE_FILES
    USE_LOCALE
    USE_LOCALE_COLLATE
    USE_LOCALE_CTYPE
    USE_LOCALE_NUMERIC
    USE_LOCALE_TIME
    USE_PERLIO
    USE_PERL_ATOF
  Built under MSWin32
  Compiled at Jan 24 2021 12:22:49
  @INC:
    C:/Strawberry/perl/site/lib
    C:/Strawberry/perl/vendor/lib
    C:/Strawberry/perl/lib
C:\Users\Owner>

[tonycoz]

It's part of #21877 and not Windows specific, though it appears to cause worse performance on Windows than on Linux (probably due to Linux not committing all allocations by default)