UTF-8 and threads #271

Michael-Adams · 2013-07-07T03:09:02Z

I haven't had any real issues from doing a "force install perl5i" show up yet. But I notice that all the Windows smoke test results on CPAN Testers share the failures listed below.

Please note that Child-0.009 installed without a hitch, not sure why it's failing your test in t/Child.t... I've attached (at the end) output of "perl -V". I'm using Citrus Perl 5.16018 and the mingw64 package they provide.

Thank you and I have been (so far) enjoying perl5i!

cpan[15]> test perl5i
Running test for module 'perl5i'
Running make for M/MS/MSCHWERN/perl5i-v2.12.0.tar.gz
Checksum for C:\Opt\bin\perl\cpan\sources\authors\id\M\MS\MSCHWERN\perl5i-v2.12.0.tar.gz ok

CPAN.pm: Building M/MS/MSCHWERN/perl5i-v2.12.0.tar.gz

Created MYMETA.yml and MYMETA.json
Creating new 'Build' script for 'perl5i' version 'v2.12.0'
Building perl5i

-->8-->8--snipped out some pod linking issue stuff-->8-->8---

MSCHWERN/perl5i-v2.12.0.tar.gz
C:\Opt\bin\perl\bin\perl.exe ./Build -- OK
Running Build test
t/ARGV.t ..................... ok
t/ARGV_twice.t ............... ok
t/CLASS.t .................... ok
t/Child.t .................... Dubious, test returned 5 (wstat 1280, 0x500)
All 1 subtests passed
t/English.t .................. ok
t/File-stat.t ................ ok
t/List-MoreUtils/all.t ....... ok
t/List-MoreUtils/any.t ....... ok
t/List-MoreUtils/false.t ..... ok
t/List-MoreUtils/mesh.t ...... ok
t/List-MoreUtils/minmax.t .... ok
t/List-MoreUtils/none.t ...... ok
t/List-MoreUtils/true.t ...... ok
t/List-MoreUtils/uniq.t ...... ok
t/List-Util/first.t .......... ok
t/List-Util/max.t ............ ok
t/List-Util/maxstr.t ......... ok
t/List-Util/min.t ............ ok
t/List-Util/minstr.t ......... ok
t/List-Util/reduce.t ......... ok
t/List-Util/shuffle.t ........ ok
t/List-Util/sum.t ............ ok
t/Meta/ISA.t ................. ok
t/Meta/checksum.t ............ ok
t/Meta/class.t ............... ok
t/Meta/id.t .................. ok
t/Meta/is-equal.t ............ ok
t/Meta/linear_isa.t .......... ok
t/Meta/methods.t ............. ok
t/Meta/reftype.t ............. ok
t/Meta/super.t ............... ok
t/Meta/symbol_table.t ........ ok
t/Want.t ..................... ok
t/alias.t .................... ok
t/as_hash.t .................. ok
t/autobox.t .................. ok
t/autodie.t .................. ok
t/autovivification.t ......... ok
t/caller.t ................... ok
t/can.t ...................... ok
t/capture.t .................. ok
t/carp.t ..................... ok
t/center.t ................... ok
t/chdir.t .................... ok
t/command_line_wrapper.t ..... 1/? "blib\script\perl5i.bat" unexpectedly returned exit value 255 at (eval 95) line 13.
at t/command_line_wrapper.t line 25

Tests were run but no plan was declared and done_testing() was not seen.

t/command_line_wrapper.t ..... Dubious, test returned 255 (wstat 65280, 0xff00)
All 2 subtests passed
t/commify.t .................. ok
t/datetime.t ................. ok
t/die.t ...................... ok
t/diff.t ..................... ok
t/dump/array.t ............... ok
t/dump/code.t ................ ok
t/dump/formats.t ............. ok
t/dump/hash.t ................ ok
t/dump/obj.t ................. ok
t/dump/scalar.t .............. ok
t/each.t ..................... eech is deprecated, use hmap instead at C:\Opt\bin\perl\cpan\build\perl5i-v2.12.0-E7oIvh\blib\lib/perl5i/2/HASH.pm line 15.
t/each.t ..................... 1/? eech is deprecated, use hmap instead at C:\Opt\bin\perl\cpan\build\perl5i-v2.12.0-E7oIvh\blib\lib/perl5i/2/HASH.pm line 15.
eech is deprecated, use hmap instead at C:\Opt\bin\perl\cpan\build\perl5i-v2.12.0-E7oIvh\blib\lib/perl5i/2/HASH.pm line 15.
eech is deprecated, use hmap instead at C:\Opt\bin\perl\cpan\build\perl5i-v2.12.0-E7oIvh\blib\lib/perl5i/2/HASH.pm line 15.
eech is deprecated, use hmap instead at C:\Opt\bin\perl\cpan\build\perl5i-v2.12.0-E7oIvh\blib\lib/perl5i/2/HASH.pm line 15.
eech is deprecated, use hmap instead at C:\Opt\bin\perl\cpan\build\perl5i-v2.12.0-E7oIvh\blib\lib/perl5i/2/HASH.pm line 15.
eech is deprecated, use hmap instead at C:\Opt\bin\perl\cpan\build\perl5i-v2.12.0-E7oIvh\blib\lib/perl5i/2/HASH.pm line 15.
eech is deprecated, use hmap instead at C:\Opt\bin\perl\cpan\build\perl5i-v2.12.0-E7oIvh\blib\lib/perl5i/2/HASH.pm line 15.
t/each.t ..................... ok
t/equal.t .................... ok
t/everything_is_an_object.t .. ok
t/flip.t ..................... ok
t/foreach.t .................. ok
t/github164.t ................ ok
t/grep.t ..................... ok
t/hash-diff.t ................ ok
t/hash-intersect.t ........... ok
t/hash-merge.t ............... ok
t/intersect.t ................ ok
t/io-handle.t ................ ok
t/is_module_name.t ........... ok
t/lexical.t .................. ok
t/list-trim.t ................ ok
t/list.t ..................... ok
t/load_together.t ............ ok
t/map.t ...................... ok
t/method_leaking.t ........... ok
t/modern_perl.t .............. ok
t/module2path.t .............. ok
t/no_indirect.t .............. ok
t/number.t ................... ok
t/perl5i.t ................... 1/? "C:\Opt\bin\perl\bin\perl.exe" unexpectedly returned exit value 255 at (eval 95) line 13.
at t/perl5i.t line 14

Tests were run but no plan was declared and done_testing() was not seen.

t/perl5i.t ................... Dubious, test returned 255 (wstat 65280, 0xff00)
All 3 subtests passed
t/pick.t ..................... ok
t/popn.t ..................... ok
t/require.t .................. ok
t/require_message.t .......... ok
t/say.t ...................... ok
t/scalar.t ................... ok
t/shiftn.t ................... ok
t/signature.t ................ ok
t/signatures.t ............... ok
t/skip.t ..................... ok
t/taint.t .................... ok
t/time_compat.t .............. ok
t/true.t ..................... ok
t/try-tiny.t ................. ok
t/uniq.t ..................... ok
t/utf8.t ..................... ok
t/version_0/00_compile.t ..... skipped: Needs Time::y2038
t/version_1/00_compile.t ..... skipped: Needs Time::y2038
t/vs_listmoreutils.t ......... ok
t/wrap.t ..................... ok
t/y2038.t .................... ok

Test Summary Report

t/Child.t (Wstat: 1280 Tests: 1 Failed: 0)
Non-zero exit status: 5
Parse errors: No plan found in TAP output
t/command_line_wrapper.t (Wstat: 65280 Tests: 2 Failed: 0)
Non-zero exit status: 255
Parse errors: No plan found in TAP output
t/perl5i.t (Wstat: 65280 Tests: 3 Failed: 0)
Non-zero exit status: 255
Parse errors: No plan found in TAP output
Files=100, Tests=1177, 59 wallclock secs ( 0.50 usr + 0.31 sys = 0.81 CPU)
Result: FAIL
Failed 3/100 test programs. 0/1177 subtests failed.
MSCHWERN/perl5i-v2.12.0.tar.gz
C:\Opt\bin\perl\bin\perl.exe ./Build test -- NOT OK
//hint// to see the cpan-testers results for installing this module, try:
reports MSCHWERN/perl5i-v2.12.0.tar.gz
Failed during this command:
MSCHWERN/perl5i-v2.12.0.tar.gz : make_test NO

Summary of my perl5 (revision 5 version 16 subversion 3) configuration:

Platform:
osname=MSWin32, osvers=6.0, archname=MSWin32-x64-multi-thread
uname=''
config_args='undef'
hint=recommended, useposix=true, d_sigaction=undef
useithreads=define, usemultiplicity=define
useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
use64bitint=define, use64bitall=undef, uselongdouble=undef
usemymalloc=n, bincompat5005=undef
Compiler:
cc='gcc', ccflags =' -s -O2 -DWIN32 -DWIN64 -DCONSERVATIVE -DPERL_RELOCATABLE_INCPUSH -DPERL_TEXTMODE_SCRIPTS -DUSE_SITECUSTOMIZE -DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS -fno-strict-aliasing -mms-bitfields',
optimize='-s -O2',
cppflags='-DWIN32'
ccversion='', gccversion='4.6.3', gccosandvers=''
intsize=4, longsize=4, ptrsize=8, doublesize=8, byteorder=12345678
d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
ivtype='long long', ivsize=8, nvtype='double', nvsize=8, Off_t='long long', lseeksize=8
alignbytes=8, prototype=define
Linker and Libraries:
ld='g++', ldflags ='-s -L"C:\Opt\bin\perl\lib\CORE" -L"C:\Opt\bin\perl\mingw64\x86_64-w64-mingw32\lib"'
libpth=C:\Opt\bin\perl\mingw64\x86_64-w64-mingw32\lib
libs=-lmoldname -lkernel32 -luser32 -lgdi32 -lwinspool -lcomdlg32 -ladvapi32 -lshell32 -lole32 -loleaut32 -lnetapi32 -luuid -lws2_32 -lmpr -lwinmm -lversion -lodbc32 -lodbccp32 -lcomctl32
perllibs=-lmoldname -lkernel32 -luser32 -lgdi32 -lwinspool -lcomdlg32 -ladvapi32 -lshell32 -lole32 -loleaut32 -lnetapi32 -luuid -lws2_32 -lmpr -lwinmm -lversion -lodbc32 -lodbccp32 -lcomctl32
libc=, so=dll, useshrplib=true, libperl=libperl516.a
gnulibc_version=''
Dynamic Linking:
dlsrc=dl_win32.xs, dlext=dll, d_dlsymun=undef, ccdlflags=' '
cccdlflags=' ', lddlflags='-mdll -s -L"C:\Opt\bin\perl\lib\CORE" -L"C:\Opt\bin\perl\mingw64\x86_64-w64-mingw32\lib"'

Characteristics of this binary (from libperl):
Compile-time options: HAS_TIMES HAVE_INTERP_INTERN MULTIPLICITY
PERLIO_LAYERS PERL_DONT_CREATE_GVSV
PERL_IMPLICIT_CONTEXT PERL_IMPLICIT_SYS
PERL_MALLOC_WRAP PERL_PRESERVE_IVUV
PERL_RELOCATABLE_INCPUSH PL_OP_SLAB_ALLOC
USE_64_BIT_INT USE_ITHREADS USE_LARGE_FILES
USE_LOCALE USE_LOCALE_COLLATE USE_LOCALE_CTYPE
USE_LOCALE_NUMERIC USE_PERLIO USE_PERL_ATOF
USE_SITECUSTOMIZE
Built under MSWin32
Compiled at Apr 3 2013 00:54:02
@inc:
C:/Opt/bin/perl/site/lib
C:/Opt/bin/perl/vendor/lib
C:/Opt/bin/perl/lib
.

exodist · 2013-07-07T04:58:12Z

I will try to answer some of these as best I can:

On Sat, Jul 6, 2013 at 8:09 PM, Michael-Adams [email protected]:

t/Child.t .................... Dubious, test returned 5 (wstat 1280, 0x500)

Fork on win32 is a farce. It is fragile and broken and occomplished using
threads, not actual process forking. Last time this was brought up I
believe it was traced to some utf8/unicode stuff in perl5i breaking forking
which in turn broke child.pm, this is why child.pm works fine in win32, but
not in perl5i-win32

t/command_line_wrapper.t ..... 1/? "blib\script\perl5i.bat" unexpectedly
returned exit value 255 at (eval 95) line 13.
at t/command_line_wrapper.t line 25

Tests were run but no plan was declared and done_testing() was not seen.

t/command_line_wrapper.t ..... Dubious, test returned 255 (wstat 65280,
0xff00)
All 2 subtests passed

No idea, needs investigation

t/each.t ..................... eech is deprecated, use hmap instead at
C:\Opt\bin\perl\cpan\build\perl5i-v2.12.0-E7oIvh\blib\lib/perl5i/2/HASH.pm
line 15.
t/each.t ..................... 1/? eech is deprecated, use hmap instead at
C:\Opt\bin\perl\cpan\build\perl5i-v2.12.0-E7oIvh\blib\lib/perl5i/2/HASH.pm
line 15.
eech is deprecated, use hmap instead at
C:\Opt\bin\perl\cpan\build\perl5i-v2.12.0-E7oIvh\blib\lib/perl5i/2/HASH.pm
line 15.
eech is deprecated, use hmap instead at
C:\Opt\bin\perl\cpan\build\perl5i-v2.12.0-E7oIvh\blib\lib/perl5i/2/HASH.pm
line 15.
eech is deprecated, use hmap instead at
C:\Opt\bin\perl\cpan\build\perl5i-v2.12.0-E7oIvh\blib\lib/perl5i/2/HASH.pm
line 15.
eech is deprecated, use hmap instead at
C:\Opt\bin\perl\cpan\build\perl5i-v2.12.0-E7oIvh\blib\lib/perl5i/2/HASH.pm
line 15.
eech is deprecated, use hmap instead at
C:\Opt\bin\perl\cpan\build\perl5i-v2.12.0-E7oIvh\blib\lib/perl5i/2/HASH.pm
line 15.
eech is deprecated, use hmap instead at
C:\Opt\bin\perl\cpan\build\perl5i-v2.12.0-E7oIvh\blib\lib/perl5i/2/HASH.pm
line 15.

These warnings are fixed as of yesterday or the day before in git, but not
in any release. The warnings are caused by an API change in
Hash::StoredIterator which perl5i uses under the hood. The newest revisions
of perl5i use the latest Hash::StoredIterator, and do not use the
deprecated functions.

t/perl5i.t ................... 1/? "C:\Opt\bin\perl\bin\perl.exe"
unexpectedly returned exit value 255 at (eval 95) line 13.
at t/perl5i.t line 14

Tests were run but no plan was declared and done_testing() was not seen.

t/perl5i.t ................... Dubious, test returned 255 (wstat 65280,
0xff00)
All 3 subtests passed

No idea on these, needs investigation.

So of the issues 1 is known, but not fixed (utf8/fork issues) Not sure if
it is fixable
Another is known and fixed
The others are not known (to me)

Michael-Adams · 2013-07-07T05:22:46Z

Ok, so...

On 2013-07-06 23:58, Chad Granum wrote:

I will try to answer some of these as best I can:

On Sat, Jul 6, 2013 at 8:09 PM, Michael-Adams
[email protected]:

t/Child.t .................... Dubious, test returned 5 (wstat 1280,
0x500)

Fork on win32 is a farce. It is fragile and broken and occomplished using
threads, not actual process forking. Last time this was brought up I
believe it was traced to some utf8/unicode stuff in perl5i breaking
forking
which in turn broke child.pm, this is why child.pm works fine in
win32, but
not in perl5i-win32

Understood, I keep tripping over the rather awful utf8 issues in perl
myself. Not that
/any/ programming language really does unicode well, especially
considering that Windows
unicode is all utf-16 internally.

t/command_line_wrapper.t ..... 1/? "blib\script\perl5i.bat" unexpectedly
returned exit value 255 at (eval 95) line 13.
at t/command_line_wrapper.t line 25

Tests were run but no plan was declared and done_testing() was not

seen.
t/command_line_wrapper.t ..... Dubious, test returned 255 (wstat 65280,
0xff00)
All 2 subtests passed

No idea, needs investigation

t/each.t .....................

These warnings are fixed as of yesterday or the day before in git, but not
in any release. The warnings are caused by an API change in
Hash::StoredIterator which perl5i uses under the hood. The newest
revisions
of perl5i use the latest Hash::StoredIterator, and do not use the
deprecated functions.

Will await the new release, and they're just depreciation warnings, so
I'm not
personally to worried about them...

t/perl5i.t ................... 1/? "C:\Opt\bin\perl\bin\perl.exe"
unexpectedly returned exit value 255 at (eval 95) line 13.
at t/perl5i.t line 14

Tests were run but no plan was declared and done_testing() was not

seen.
t/perl5i.t ................... Dubious, test returned 255 (wstat 65280,
0xff00)
All 3 subtests passed

No idea on these, needs investigation.

So of the issues 1 is known, but not fixed (utf8/fork issues) Not sure if
it is fixable
Another is known and fixed
The others are not known (to me)

—
Reply to this email directly or view it on GitHub
#271 (comment).

Thanks! If I can help by testing or supplying any other info for any of
these 3
outstanding issues, just let me know.

Michael Adams
SQL/Perl/VBA/C??/Java

exodist · 2013-07-07T22:04:31Z

On Sat, Jul 6, 2013 at 10:22 PM, Michael Adams [email protected]:

Thanks! If I can help by testing or supplying any other info for any of
these 3
outstanding issues, just let me know.

I am not sure if perl5i has any windows dev's devoting any effort to it. If
you wanted to to try and debug/fix these issues I am sure it would be
appreciated by everyone involved.

Michael-Adams · 2013-07-08T00:08:33Z

On 2013-07-07 17:04, Chad Granum wrote:

On Sat, Jul 6, 2013 at 10:22 PM, Michael Adams
[email protected]:

Thanks! If I can help by testing or supplying any other info for any of
these 3
outstanding issues, just let me know.

I am not sure if perl5i has any windows dev's devoting any effort to
it. If
you wanted to to try and debug/fix these issues I am sure it would be
appreciated by everyone involved.

—
Reply to this email directly or view it on GitHub
#271 (comment).

Well, I've started on the t/Child.t issue, and it /is/ related to utf8
processing:

use strict;
use warnings;
use utf8::all;
use Child;

my $child = Child->new( sub {
my $self = shift;
$self->say( "Have self" );
$self->say( "parent: " . $self->pid );
my $in = $self->read();
$self->say( $in );
},
pipe => 1
);

my $proc = $child->start;

It dies in Child.pm on line 61: if ( my $pid = fork() ) {

If one comments out the "use utf8::all;" line, it goes to completion.

Not quite sure where to go from here... It's looking like maybe a p5p
issue as utf8::all breaks fork on win32?
Especially as I don't see where (in utf8::all source) it would have a
bad interaction.

Michael Adams
SQL/Perl/VBA/C??/Java

exodist · 2013-07-08T00:19:21Z

http://www.nntp.perl.org/group/perl.perl5.porters/2012/12/msg196821.html

On Sun, Jul 7, 2013 at 5:08 PM, Michael Adams [email protected]:

On 2013-07-07 17:04, Chad Granum wrote:

On Sat, Jul 6, 2013 at 10:22 PM, Michael Adams
[email protected]:

Thanks! If I can help by testing or supplying any other info for any of
these 3
outstanding issues, just let me know.

I am not sure if perl5i has any windows dev's devoting any effort to
it. If
you wanted to to try and debug/fix these issues I am sure it would be
appreciated by everyone involved.

—
Reply to this email directly or view it on GitHub
#271 (comment).

Well, I've started on the t/Child.t issue, and it /is/ related to utf8
processing:

|use strict;
use warnings;
use utf8::all;
use Child;

my $child = Child->new( sub {
my $self = shift;
$self->say( "Have self" );
$self->say( "parent: " . $self->pid );
my $in = $self->read();
$self->say( $in );
},
pipe => 1
);

my $proc = $child->start;

|It dies in Child.pm on line 61: | if ( my $pid = fork() ) {

If one comments out the "use utf8::all;" line, it goes to completion.
|
Not quite sure where to go from here... It's looking like maybe a p5p
issue as utf8::all breaks fork on win32?
Especially as I don't see where (in utf8::all source) it would have a
bad interaction.

Michael Adams
SQL/Perl/VBA/C??/Java

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/271#issuecomment-20580327
.

exodist · 2013-07-08T00:22:56Z

https://rt.perl.org/rt3//Public/Bug/Display.html?id=31923

On Sun, Jul 7, 2013 at 5:19 PM, Chad Granum [email protected] wrote:

http://www.nntp.perl.org/group/perl.perl5.porters/2012/12/msg196821.html

On Sun, Jul 7, 2013 at 5:08 PM, Michael Adams [email protected]:

On 2013-07-07 17:04, Chad Granum wrote:

On Sat, Jul 6, 2013 at 10:22 PM, Michael Adams
[email protected]:

Thanks! If I can help by testing or supplying any other info for any
of
these 3
outstanding issues, just let me know.

I am not sure if perl5i has any windows dev's devoting any effort to
it. If
you wanted to to try and debug/fix these issues I am sure it would be
appreciated by everyone involved.

—
Reply to this email directly or view it on GitHub
#271 (comment).

Well, I've started on the t/Child.t issue, and it /is/ related to utf8
processing:

|use strict;
use warnings;
use utf8::all;
use Child;

my $child = Child->new( sub {
my $self = shift;
$self->say( "Have self" );
$self->say( "parent: " . $self->pid );
my $in = $self->read();
$self->say( $in );
},
pipe => 1
);

my $proc = $child->start;

|It dies in Child.pm on line 61: | if ( my $pid = fork() ) {

If one comments out the "use utf8::all;" line, it goes to completion.
|
Not quite sure where to go from here... It's looking like maybe a p5p
issue as utf8::all breaks fork on win32?
Especially as I don't see where (in utf8::all source) it would have a
bad interaction.

Michael Adams
SQL/Perl/VBA/C??/Java

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/271#issuecomment-20580327
.

exodist · 2013-07-08T00:26:32Z

This really is not a perl5i bug. It is a bug with threads and utf8. It is
not a problem on linux (usually) because people rarely use perl threads in
linux. However on windows, forking is emulated using threads.

Perl5i has a choice between utf8-all and broken fork, or working fork w/o
utf8.

However, If utf8all is scoped, it might be possible to have Child.pm
actively turn it off before calling fork()....

On Sun, Jul 7, 2013 at 5:22 PM, Chad Granum [email protected] wrote:

https://rt.perl.org/rt3//Public/Bug/Display.html?id=31923

On Sun, Jul 7, 2013 at 5:19 PM, Chad Granum [email protected] wrote:

http://www.nntp.perl.org/group/perl.perl5.porters/2012/12/msg196821.html

On Sun, Jul 7, 2013 at 5:08 PM, Michael Adams [email protected]:

On 2013-07-07 17:04, Chad Granum wrote:

On Sat, Jul 6, 2013 at 10:22 PM, Michael Adams
[email protected]:

Thanks! If I can help by testing or supplying any other info for any
of
these 3
outstanding issues, just let me know.

I am not sure if perl5i has any windows dev's devoting any effort to
it. If
you wanted to to try and debug/fix these issues I am sure it would be
appreciated by everyone involved.

—
Reply to this email directly or view it on GitHub
#271 (comment).

Well, I've started on the t/Child.t issue, and it /is/ related to utf8
processing:

|use strict;
use warnings;
use utf8::all;
use Child;

my $child = Child->new( sub {
my $self = shift;
$self->say( "Have self" );
$self->say( "parent: " . $self->pid );
my $in = $self->read();
$self->say( $in );
},
pipe => 1
);

my $proc = $child->start;

|It dies in Child.pm on line 61: | if ( my $pid = fork() ) {

If one comments out the "use utf8::all;" line, it goes to completion.
|
Not quite sure where to go from here... It's looking like maybe a p5p
issue as utf8::all breaks fork on win32?
Especially as I don't see where (in utf8::all source) it would have a
bad interaction.

Michael Adams
SQL/Perl/VBA/C??/Java

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/271#issuecomment-20580327
.

exodist · 2013-07-08T00:29:23Z

Looks like it is lexical, I am going to see if I can find a solution.

On Sun, Jul 7, 2013 at 5:26 PM, Chad Granum [email protected] wrote:

This really is not a perl5i bug. It is a bug with threads and utf8. It is
not a problem on linux (usually) because people rarely use perl threads in
linux. However on windows, forking is emulated using threads.

Perl5i has a choice between utf8-all and broken fork, or working fork w/o
utf8.

However, If utf8all is scoped, it might be possible to have Child.pm
actively turn it off before calling fork()....

On Sun, Jul 7, 2013 at 5:22 PM, Chad Granum [email protected] wrote:

https://rt.perl.org/rt3//Public/Bug/Display.html?id=31923

On Sun, Jul 7, 2013 at 5:19 PM, Chad Granum [email protected] wrote:

http://www.nntp.perl.org/group/perl.perl5.porters/2012/12/msg196821.html

On Sun, Jul 7, 2013 at 5:08 PM, Michael Adams [email protected]:

On 2013-07-07 17:04, Chad Granum wrote:

On Sat, Jul 6, 2013 at 10:22 PM, Michael Adams
[email protected]:

Thanks! If I can help by testing or supplying any other info for
any of
these 3
outstanding issues, just let me know.

I am not sure if perl5i has any windows dev's devoting any effort to
it. If
you wanted to to try and debug/fix these issues I am sure it would be
appreciated by everyone involved.

—
Reply to this email directly or view it on GitHub
#271 (comment).

Well, I've started on the t/Child.t issue, and it /is/ related to utf8
processing:

|use strict;
use warnings;
use utf8::all;
use Child;

my $child = Child->new( sub {
my $self = shift;
$self->say( "Have self" );
$self->say( "parent: " . $self->pid );
my $in = $self->read();
$self->say( $in );
},
pipe => 1
);

my $proc = $child->start;

|It dies in Child.pm on line 61: | if ( my $pid = fork() ) {

If one comments out the "use utf8::all;" line, it goes to completion.
|
Not quite sure where to go from here... It's looking like maybe a p5p
issue as utf8::all breaks fork on win32?
Especially as I don't see where (in utf8::all source) it would have a
bad interaction.

Michael Adams
SQL/Perl/VBA/C??/Java

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/271#issuecomment-20580327
.

Michael-Adams · 2013-07-08T01:14:54Z

On 2013-07-07 19:29, Chad Granum wrote:

Looks like it is lexical, I am going to see if I can find a solution.

On Sun, Jul 7, 2013 at 5:26 PM, Chad Granum [email protected] wrote:

This really is not a perl5i bug. It is a bug with threads and utf8.
It is
not a problem on linux (usually) because people rarely use perl
threads in
linux. However on windows, forking is emulated using threads.

Perl5i has a choice between utf8-all and broken fork, or working
fork w/o
utf8.

However, If utf8all is scoped, it might be possible to have Child.pm
actively turn it off before calling fork()....
Just spent a fair amount of time researching it as well based on your
first hint to me.

Traipsed through the Perl source code involved, took Ibuprofen...

If you can turn the utf8 off for Child.pm, it may be the only fix until
p5p figures out how to make utf8 and threads play nice together. But
since the bug is open and was registered against v5.8.3!, I'm not sure
they /can/ fix it without a huge amount of work as that section of Perl
source code is rather a mess -- It makes me think, just looking at it,
that Perl's threads aren't quite "production" ready -- even after all
this time.

There seems to be more than one possibly dodgy stack
manipulation involved in emulating fork(), along with possibly more than
one initialization sequencing issue of the catch-22 variety.

Later on this week I'll try digging into the other 2 items.

Michael Adams
SQL/Perl/VBA/C??/Java

exodist · 2013-07-08T01:33:53Z

I was not able to find a way to lexically tun it off in Child. I was able
to reproduce the failure, but not eliminate it :-(

I think perl5i should probably not include utf8 on windows. I say this
because people can always load utf8 if they need it, but they cannot turn
it off. So if anyone needs to use forking on windows and wants to use
perl5i they cannot.

At the same time I am hesitant to suggest we remove utf8-all because if
anyone uses forking on win32 they are already making a mistake.

We could also skip the child.pm tests in win32 under perl5i, but then
people will not know it is broken and will report the issue again and again.

On Sun, Jul 7, 2013 at 6:14 PM, Michael Adams [email protected]:

On 2013-07-07 19:29, Chad Granum wrote:

Looks like it is lexical, I am going to see if I can find a solution.

On Sun, Jul 7, 2013 at 5:26 PM, Chad Granum [email protected] wrote:

This really is not a perl5i bug. It is a bug with threads and utf8.
It is
not a problem on linux (usually) because people rarely use perl
threads in
linux. However on windows, forking is emulated using threads.

Perl5i has a choice between utf8-all and broken fork, or working
fork w/o
utf8.

However, If utf8all is scoped, it might be possible to have Child.pm
actively turn it off before calling fork()....
Just spent a fair amount of time researching it as well based on your
first hint to me.

Traipsed through the Perl source code involved, took Ibuprofen...

If you can turn the fut8 off for Child.pm, it may be the only fix until
p5p figures out how to make fut8 and threads play nice together. But
since the bug is open and was registered against v5.83!, I'm not sure
they /can/ fix it without a huge amount of work as that section of Perl
source code is rather a mess -- It makes me think, just looking at it,
that Perl's threads aren't quite "production" ready -- even after all
this time. There seems to be more than one possibly dodgy stack
manipulation involved in emulating fork(), along with possibly more than
one initialization sequencing issue of the catch-22 variety.

Later on this week I'll try digging into the other 2 items.

Michael Adams
SQL/Perl/VBA/C??/Java

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/271#issuecomment-20581442
.

Michael-Adams · 2013-07-08T02:01:58Z

So the possibilities are a "$^O" test in perl5i startup code that :

Disables utf8 on windows with a strong note in a "CAVEATS" section of the POD (not "BUGS" as it's not a perl5i bug) noting that utf8 is not turned on by perl5i due to this issue and also mentioning that fork() is not really recommended on windows.
Or keeps utf8 on windows but not Child (as fork() is contraindicated on win32)... with a similar strong note, unless it's just fork() that's borked by utf8, not threads per se... (need to test that...) and Child on windows can be reworked to do its own explicit fork() emulation via threads (most probably way too much work for too little gain...).

exodist · 2013-07-08T02:07:53Z

Actually. Maybe a better solution in perl5i would be to override
CORE::GLOBAL::Fork to throw an exception explaining the situation and that
perl5i breaks fork on windows. This will catch uses by Child.pm (though
Child() can also be overriden). The Child tests for perl5i can skip in
win32.

On Sun, Jul 7, 2013 at 7:01 PM, Michael Adams [email protected]:

So the possibilities are a "$^O" test in perl5i startup code that :

Disables utf8 on windows with a strong note in a "CAVEATS" section of
the POD (not "BUGS" as it's not a perl5i bug) noting that utf8 is not
turned on by perl5i due to this issue and also mentioning that fork() is
not really recommended on windows.

Or keeps utf8 on windows but not Child (as fork() is contraindicated
on win32)... with a similar strong note, unless it's just fork() that's
borked by utf8, not threads per se... (need to test that...) and Child on
windows can be reworked to do its own explicit fork() emulation via threads
(most probably way too much work for too little gain...).

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/271#issuecomment-20582343
.

Michael-Adams · 2013-07-08T02:29:08Z

I think I'd +1 the CORE::GLOBAL::Fork solution for win32 installs of perl5i. And of course I must'a been all goofy from reading "sv.c"; threads and utf8 are contraindicated on seemingly all systems...

exodist · 2013-07-08T03:13:58Z

I would prefer to have Schwern and others weigh in on this, I don't think
it is a decision you and I should make :-)

On Sun, Jul 7, 2013 at 7:29 PM, Michael Adams [email protected]:

I think I'd +1 the CORE::GLOBAL::Fork solution for win32 installs of
perl5i. And of course I must'a been all goofy from reading "sv.c"; threads
and utf8 are contraindicated on seemingly all systems...

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/271#issuecomment-20582868
.

Michael-Adams · 2013-07-08T03:29:55Z

On 2013-07-07 22:13, Chad Granum wrote:

I would prefer to have Schwern and others weigh in on this, I don't think
it is a decision you and I should make :-)

On Sun, Jul 7, 2013 at 7:29 PM, Michael Adams
[email protected]:

I think I'd +1 the CORE::GLOBAL::Fork solution for win32 installs of
perl5i. And of course I must'a been all goofy from reading "sv.c";
threads
and utf8 are contraindicated on seemingly all systems...

—
Reply to this email directly or view it on
GitHubhttps://github.com//issues/271#issuecomment-20582868
.

—
Reply to this email directly or view it on GitHub
#271 (comment).

True, thus the /"I t//hink I'd +1"/part:) and anyway Schwern being more
experienced than, well me definitely, may have an even better solution.
Good night and have a good day!

Michael Adams
SQL/Perl/VBA/C??/Java

schwern · 2016-08-08T16:48:08Z

I do not relish having some features that only work on some platforms.

I chose Child for perl5i not just because it provides a nice interface to fork, but because it provides a nice abstraction for interprocess work. Looking at the Child interface, I wonder if it can be rewritten on Windows to use threads (which hopefully are less buggy when not trying to be fork) or native Windows IPC.

exodist · 2016-08-08T17:19:43Z

Making Child use threads on windows would be sane, and would solve many problems. That said it will not solve this problem :-(

https://rt.perl.org/Public/Bug/Display.html?id=31923 is the rt ticket that covers the problem in perl itself. Basically "PerlIO::encoding isn't thread safe."

There is a workaround I use in Test2:
https://metacpan.org/pod/Test2::Plugin::UTF8#NOTES

Which uses binmode(HANDLE, ':utf8'); This is not as good as :encoding(utf-8) which is more strict, but on the other hand it actually works, which I consider a bonus.

Note that linux also has this problem, the reason we only notice on windows is because fork on windows is actually threads under the hood. If you used perl5i with threads you would likely encounter the same issue on linux,

exodist · 2016-08-08T17:24:51Z

Hmm, another option would be to require a newer version of perl, it appears the issue was fixed in blead last October, but I think that means depending on 5.24.0+.

schwern · 2016-08-08T20:58:13Z

If threads + :encoding(utf8) is as broken as that perlbug thread says, I think I'll have perl5i use just :utf8.

exodist · 2016-08-08T21:01:14Z

@schwern I agree that is probably the most sane option.

schwern · 2016-08-09T17:42:26Z

doherty/utf8-all#43 should fix this problem. Once it's in utf8::all we'll depend on the new version.

schwern · 2016-08-10T19:19:33Z

Damn, that didn't fix it. But it got us further along.
https://ci.appveyor.com/project/schwern/perl5i/build/7

Free to wrong pool 32180a0 not 859b10 at C:/strawberry/perl/site/lib/Child.pm line 86.
t/Child.t .................... 
Dubious, test returned 5 (wstat 1280, 0x500)
All 5 subtests passed

I've seen that "free to wrong pool" before when trying to free memory using Perl's free() that was allocated using the system's malloc().
http://www.perlmonks.org/?node_id=717224

schwern mentioned this issue Aug 8, 2016

threads + utf8::all == boom doherty/utf8-all#42

Closed

schwern changed the title ~~Install errors (v2.12.0) on perl5.16.3 / Windows 7~~ UTF-8 and threads Aug 9, 2016

schwern added the bug label Aug 9, 2016

schwern closed this as completed in bdb2936 Aug 10, 2016

schwern reopened this Aug 10, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

UTF-8 and threads #271

UTF-8 and threads #271

Michael-Adams commented Jul 7, 2013

exodist commented Jul 7, 2013

Tests were run but no plan was declared and done_testing() was not seen.

Tests were run but no plan was declared and done_testing() was not seen.

Michael-Adams commented Jul 7, 2013

Tests were run but no plan was declared and done_testing() was not

Tests were run but no plan was declared and done_testing() was not

exodist commented Jul 7, 2013

Michael-Adams commented Jul 8, 2013

exodist commented Jul 8, 2013

exodist commented Jul 8, 2013

exodist commented Jul 8, 2013

exodist commented Jul 8, 2013

Michael-Adams commented Jul 8, 2013

exodist commented Jul 8, 2013

Michael-Adams commented Jul 8, 2013

exodist commented Jul 8, 2013

Disables utf8 on windows with a strong note in a "CAVEATS" section of
the POD (not "BUGS" as it's not a perl5i bug) noting that utf8 is not
turned on by perl5i due to this issue and also mentioning that fork() is
not really recommended on windows.

Michael-Adams commented Jul 8, 2013

exodist commented Jul 8, 2013

Michael-Adams commented Jul 8, 2013

schwern commented Aug 8, 2016

exodist commented Aug 8, 2016

exodist commented Aug 8, 2016

schwern commented Aug 8, 2016

exodist commented Aug 8, 2016

schwern commented Aug 9, 2016

schwern commented Aug 10, 2016

UTF-8 and threads #271

UTF-8 and threads #271

Comments

Michael-Adams commented Jul 7, 2013

Tests were run but no plan was declared and done_testing() was not seen.

Tests were run but no plan was declared and done_testing() was not seen.

Test Summary Report

exodist commented Jul 7, 2013

Tests were run but no plan was declared and done_testing() was not seen.

Tests were run but no plan was declared and done_testing() was not seen.

Michael-Adams commented Jul 7, 2013

Tests were run but no plan was declared and done_testing() was not

Tests were run but no plan was declared and done_testing() was not

exodist commented Jul 7, 2013

Michael-Adams commented Jul 8, 2013

exodist commented Jul 8, 2013

exodist commented Jul 8, 2013

exodist commented Jul 8, 2013

exodist commented Jul 8, 2013

Michael-Adams commented Jul 8, 2013

exodist commented Jul 8, 2013

Michael-Adams commented Jul 8, 2013

exodist commented Jul 8, 2013

Disables utf8 on windows with a strong note in a "CAVEATS" section of the POD (not "BUGS" as it's not a perl5i bug) noting that utf8 is not turned on by perl5i due to this issue and also mentioning that fork() is not really recommended on windows.

Michael-Adams commented Jul 8, 2013

exodist commented Jul 8, 2013

Michael-Adams commented Jul 8, 2013

schwern commented Aug 8, 2016

exodist commented Aug 8, 2016

exodist commented Aug 8, 2016

schwern commented Aug 8, 2016

exodist commented Aug 8, 2016

schwern commented Aug 9, 2016

schwern commented Aug 10, 2016

Disables utf8 on windows with a strong note in a "CAVEATS" section of
the POD (not "BUGS" as it's not a perl5i bug) noting that utf8 is not
turned on by perl5i due to this issue and also mentioning that fork() is
not really recommended on windows.