Build entire Phobos with -preview=dip1000

[wilzbach]

With dlang/dmd#9476, this could work now ...

[dlang-bot]

Thanks for your pull request, @wilzbach!

Bugzilla references

Your PR doesn't reference any Bugzilla issue.

If your PR contains non-trivial changes, please reference a Bugzilla issue or create a manual changelog.

Testing this PR locally

If you don't have a local development environment setup, you can use Digger to test this PR:

dub fetch digger
dub run digger -- build "master + phobos#6931"

[thewilsonator]

generated/linux/debug/64/publictests/std_exception.d(1251): Error: scope variable p may not be copied into allocated memory

[wilzbach]

That could be due to the public unittest checker dropping the unittest attributes, but the linking errors on the Auto-Tester are still there and real.

[WalterBright]

I'm confused. Why did this work with dip1000.mak with it all set to dip1000, and this fails?

[WalterBright]

generated/linux/debug/64/publictests/std_exception.d(1251): Error: scope variable `p` may not be copied into allocated memory
posix.mak:628: recipe for target 'std/exception.publictests' failed

What is std_exception.d generated from?

[wilzbach]

I'm confused. Why did this work with dip1000.mak with it all set to dip1000, and this fails?

It never worked. It only worked for the unittest target because we compiled all Phobos with it, there that's why there are no linker errors there. We never compiled Phobos as a library with dip1000, because as I have been constantly saying it never worked.

[thewilsonator]

What is std_exception.d generated from?

std.exception, I'm confused as to why that gives that error considering the unittest is explicitly @system. see also https://issues.dlang.org/show_bug.cgi?id=18637

[WalterBright]

Why would std.exception fail when compiled as a library but work compiled as a unittest?

[WalterBright]

I'm also confused:

@system unittest
{
    int i;
    int* p = &i;
    int*[] slicep = [p]; // the line in question
}

compiles without error with and without -dip1000

[WalterBright]

So does compiling std.exception with -dip1000 - no errors.

[wilzbach]

I would start looking at the linker errors on the Auto-Tester which show that the recent exclusion of the inferred return in the mangling didn't cover everything:

../generated/freebsd/release/32/examples/impvisitor.o: In function `_D3std6format__T14formattedWriteTSQBgQBf8NoOpSinkTaTAyaTkZQBqFNaNfQBhMxAaQvkZk':
examples/impvisitor.d:(.text._D3std6format__T14formattedWriteTSQBgQBf8NoOpSinkTaTAyaTkZQBqFNaNfQBhMxAaQvkZk+0x2b): undefined reference to `_D3std6format__T10FormatSpecTaZQp6__ctorMFNaNbNcNiNfxAaZSQCdQCc__TQByTaZQCe'
../generated/freebsd/release/32/examples/impvisitor.o: In function `_D3std6format__T11formatRangeTSQBdQBc8NoOpSinkTAyaTaZQBlFNaNfKQBgKQtMKxSQCsQCr__T10FormatSpecTaZQpZv':
examples/impvisitor.d:(.text._D3std6format__T11formatRangeTSQBdQBc8NoOpSinkTAyaTaZQBlFNaNfKQBgKQtMKxSQCsQCr__T10FormatSpecTaZQpZv+0x2cf): undefined reference to `_D3std6format__T10FormatSpecTaZQp6__ctorMFNaNbNcNiNfxAaZSQCdQCc__TQByTaZQCe'
../generated/freebsd/release/32/examples/impvisitor.o: In function `_D3std6format__T14formattedWriteTSQBgQBf8NoOpSinkTaTkZQBmFNaNfKQBeMxAakZk':
examples/impvisitor.d:(.text._D3std6format__T14formattedWriteTSQBgQBf8NoOpSinkTaTkZQBmFNaNfKQBeMxAakZk+0x2b): undefined reference to `_D3std6format__T10FormatSpecTaZQp6__ctorMFNaNbNcNiNfxAaZSQCdQCc__TQByTaZQCe'
../generated/freebsd/release/32/examples/impvisitor.o: In function `_D3std6format__T11guessLengthTaTAyaZQuFNaNfQlZk':
examples/impvisitor.d:(.text._D3std6format__T11guessLengthTaTAyaZQuFNaNfQlZk+0x37): undefined reference to `_D3std6format__T10FormatSpecTaZQp6__ctorMFNaNbNcNiNfxAaZSQCdQCc__TQByTaZQCe'
../generated/freebsd/release/32/examples/impvisitor.o: In function `_D3std6format__T14formattedWriteTSQBg5array__T8AppenderTAyaZQoTaTQjTkZQCcFNaNfKQBuMxAaQBekZk':
examples/impvisitor.d:(.text._D3std6format__T14formattedWriteTSQBg5array__T8AppenderTAyaZQoTaTQjTkZQCcFNaNfKQBuMxAaQBekZk+0x2b): undefined reference to `_D3std6format__T10FormatSpecTaZQp6__ctorMFNaNbNcNiNfxAaZSQCdQCc__TQByTaZQCe'
../generated/freebsd/release/32/examples/impvisitor.o:examples/impvisitor.d:(.text._D3std6format__T11formatRangeTSQBd5array__T8AppenderTAyaZQoTQhTaZQBxFNaNfKQBsKQzMKxSQDeQDd__T10FormatSpecTaZQpZv+0x2cf): more undefined references to `_D3std6format__T10FormatSpecTaZQp6__ctorMFNaNbNcNiNfxAaZSQCdQCc__TQByTaZQCe' follow
cc: error: linker command failed with exit code 1 (use -v to see invocation)
Error: linker exited with status 1

[wilzbach]

I'm confused as to why that gives that error considering the unittest is explicitly @System

Nope, they get dropped by the unittest parser (the external tool, not the compiler). They get dropped because for user-facing public examples their unittest attributes shouldn't matter.

Anyhow, how is my previous reply not explanatory?

due to the public unittest checker dropping the unittest attributes, but the linking errors on the Auto-Tester are still there and real.

[WalterBright]

Nope, they get dropped by the unittest parser (the external tool, not the compiler). They get dropped because for user-facing public examples their unittest attributes shouldn't matter.

What do you suggest be done to correct the failure?

[WalterBright]

I would start looking at the linker errors on the Auto-Tester which show that the recent exclusion of the inferred return in the mangling didn't cover everything:

It's kinda hard to figure out what is going on, since I don't know what compiled with what is linking with what compiled with what. The linker error says what wasn't found, but doesn't say what is in the library that should match it.

If you could find a file that compiled with different mangling depending on -dip1000 on or off, that would be sufficient for me to find the problem.

[wilzbach]

If you could find a file that compiled with different mangling depending on -dip1000 on or off, that would be sufficient for me to find the problem.

How about this bug report from a year ago?

https://issues.dlang.org/show_bug.cgi?id=18820

Alternatively, to reproduce the error on auto-tester:

1. Checkout this PR or do alternatively do this:

make -f posix.mak DFLAGS=-previw=dip1000 all

2. The test that's failing on the auto-tester is this one:

cd ~/dland/dmd/src
~/dlang/dmd/generated/linux/release/64/dmd -i -J.. examples/impvisitor.d

So as mentioned any more complex combination of mixing object files compiled with and without -preview=dip1000 still results in linker errors.

The specific example from the auto-tester can be reduced to this file:

void main() {
    import std.file;
    auto input = readText("foo");
}

Maybe that's because -preview=dip1000 silently treats in different?

[WalterBright]

https://issues.dlang.org/show_bug.cgi?id=18820 is working.

[WalterBright]

See dlang/dmd#9508 for solution to at least some of these issues.

[WalterBright]

@wilzbach can you ping the autotester to restart testing, now that 9508 has been pulled?

[thewilsonator]

Just force push or rebase.

[WalterBright]

Just force push or rebase.

It's not my PR, so I don't think I can do that.

[wilzbach]

Sure you can. All maintainers can push to all PRs. Anyhow, as you all know the PRs at the Auto-Tester get automatically restarted whenever master moves forward. Have a look at the build logs and datestamps. It's from today and yup still failing.

[thewilsonator]

For reference:

std/regex/internal/parser.d:984: undefined reference to '_D3std3uni__T5StackTkZQj3topMFNaNbNcNdNiNfZk'

[wilzbach]

Reduced the issue to https://issues.dlang.org/show_bug.cgi?id=19857

[wilzbach]

Now it fails due to https://issues.dlang.org/show_bug.cgi?id=19873

[wilzbach]

PR to the publictests extractor -> dlang/tools#369
This allows moving on to the probably next issue without waiting for the DMD issue to be fixed.

[wilzbach]

Incredible. This is actually passing now 🎉

CC @thewilsonator @WalterBright

[thewilsonator]

This should probably have a changelog entry at some point.

[wilzbach]

This should probably have a changelog entry at some point.

I can add a quick one, but as pointed out at DConf, this is the easy part of the dip1000 transition. Making all 1500 dub packages build with it, will require a lot of effort and energy plus very likely deprecation messages.

[wilzbach]

This should probably have a changelog entry at some point.

I can add a quick one,

-> #7005