Pretty print carton test output

[carson-katri]

Generalized the ProcessRunner to take a ProcessOutputParser (currently only DiagnosticsParser and TestsParser conform).

Here's a sample of passing & failing test cases:

And here's what the raw output of swift test would be:

Test Suite 'All tests' started at 2020-10-26 21:33:57.453
Test Suite 'testBundle.xctest' started at 2020-10-26 21:33:57.453
Test Suite 'Test' started at 2020-10-26 21:33:57.454
Test Case 'Test.testTrivial' started at 2020-10-26 21:33:57.460
/Users/carson/.carton/carton/TestApp/Tests/Tests/Test.swift:21: error: Test.testTrivial : XCTAssertEqual failed: ("Hello, world") is not equal to ("Hello world") - 
/Users/carson/.carton/carton/TestApp/Tests/Tests/Test.swift:22: error: Test.testTrivial : XCTAssertGreaterThan failed: ("5") is not greater than ("5") - 
/Users/carson/.carton/carton/TestApp/Tests/Tests/Test.swift:23: error: Test.testTrivial : XCTAssertLessThan failed: ("5") is not less than ("4") - 
/Users/carson/.carton/carton/TestApp/Tests/Tests/Test.swift:24: error: Test.testTrivial : XCTAssertLessThanOrEqual failed: ("5") is greater than ("4") - 
/Users/carson/.carton/carton/TestApp/Tests/Tests/Test.swift:25: error: Test.testTrivial : XCTAssertGreaterThanOrEqual failed: ("5") is less than ("6") - 
Test Case 'Test.testTrivial' failed (0.001 seconds)
Test Suite 'Test' failed at 2020-10-26 21:33:57.461
	 Executed 2 tests, with 6 failures (1 unexpected) in 0.007 (0.007) seconds
Test Suite 'testBundle.xctest' failed at 2020-10-26 21:33:57.461
	 Executed 2 tests, with 6 failures (1 unexpected) in 0.007 (0.007) seconds
Test Suite 'All tests' failed at 2020-10-26 21:33:57.461
	 Executed 2 tests, with 6 failures (1 unexpected) in 0.007 (0.007) seconds

[MaxDesiatov]

This is marvelous! 👏 I myself lost all hope for parsing XCTest output long time ago, and that's the reason I hacked an XCTest JSON output formatter in our development snapshots (but it didn't land in our 5.3 due to lack of time). This amazing work means I can revert the JSON output formatter, as it had almost no chances to land upstream in its current form. Even though XCTest JSON output is more elegant in theory.

Maybe this output parser could become a separate library at some point? But I'm surely fine with merging it now here and battle-testing for a bit to make sure we can publish it later as a separate library for other people to rely on.

[MaxDesiatov]

Please add a doc comment for this property.

[carson-katri]

Yeah it'd be nice to separate these from carton. Hopefully we can get Json output in the future (and for compiler output too 🤞)

[MaxDesiatov]

The compiler can already output parseable diagnostics, I'm not sure about JSON, but bitstream output is available for sure, and now can be parsed in pure Swift thanks to swiftlang/swift-tools-support-core#120 and swiftlang/swift-tools-support-core#145.

[carson-katri]

Oh, awesome!

[carson-katri]

I tried to implement the DiagnosticsParser with TSCUtility.SerializedDiagnostics, but it seems to fall apart if there are fixits... I kept getting the error unexpectedSubblock.

[MaxDesiatov]

Thanks 👍

[MaxDesiatov]

Seems like TSC support for diagnostics in the bitcode format is unfinished 😕