Fix the start/end byte positions in the compiler JSON output #65074

Rantanen · 2019-10-03T22:03:13Z

Track the changes made during normalization in the SourceFile and use this information to correct the start_byte and end_byte fields in the JSON output.

This should ensure the start/end byte fields can be used to index the original file, even if Rust normalized the source code for parsing purposes. Both CRLF to LF and BOM removal are handled with this one.

The rough plan was discussed with @matklad in rust-lang/rustfix#176 - although I ended up going with u32 offset tracking so I wouldn't need to deal with u32 + i32 arithmetics when applying the offset to the span byte positions.

Fixes #65029

rust-highfive · 2019-10-03T22:03:17Z

r? @petrochenkov

(rust_highfive has picked a reviewer for you, use r? to override)

Rantanen · 2019-10-03T22:38:30Z

I added the normalized_pos field in SourceFile next to the existing position based bits (multibyte_chars and non_narrow_chars) as it felt conceptually similar to those fields. Not sure if I should have added it as the last field for chronological reasons.

This got a bit more confusing when implementing encode/decode for the type as there I didn't want to change the existing field indices (not sure if there's some compatibility requirements with those functions), so there I added the normalized_pos as the last field in field id order.

(I also managed to sneak in a typo with my last refactoring that fails the build, I'll be rebasing this ~~in a moment~~ in the morning once my build goes through.)

rust-highfive · 2019-10-03T23:05:15Z

The job x86_64-gnu-llvm-6.0 of your PR failed (pretty log, raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

2019-10-03T22:45:50.7129455Z ##[command]git remote add origin https://github.com/rust-lang/rust
2019-10-03T22:45:50.7361669Z ##[command]git config gc.auto 0
2019-10-03T22:45:50.7487592Z ##[command]git config --get-all http.https://github.com/rust-lang/rust.extraheader
2019-10-03T22:45:50.7494507Z ##[command]git config --get-all http.proxy
2019-10-03T22:45:50.7619058Z ##[command]git -c http.extraheader="AUTHORIZATION: basic ***" fetch --force --tags --prune --progress --no-recurse-submodules --depth=2 origin +refs/heads/*:refs/remotes/origin/* +refs/pull/65074/merge:refs/remotes/pull/65074/merge
---
2019-10-03T22:53:00.2779451Z    Compiling serde_json v1.0.40
2019-10-03T22:53:02.0718808Z    Compiling tidy v0.1.0 (/checkout/src/tools/tidy)
2019-10-03T22:53:13.2074542Z     Finished release [optimized] target(s) in 1m 28s
2019-10-03T22:53:13.2150912Z tidy check
2019-10-03T22:53:14.1294428Z tidy error: /checkout/src/libsyntax/tests.rs: too many trailing newlines (2)
2019-10-03T22:53:15.2836883Z some tidy checks failed
2019-10-03T22:53:15.2843409Z 
2019-10-03T22:53:15.2843409Z 
2019-10-03T22:53:15.2845972Z command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/tidy" "/checkout/src" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "--no-vendor"
2019-10-03T22:53:15.2846148Z 
2019-10-03T22:53:15.2846173Z 
2019-10-03T22:53:15.2854873Z failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test src/tools/tidy
2019-10-03T22:53:15.2854971Z Build completed unsuccessfully in 0:01:31
2019-10-03T22:53:15.2854971Z Build completed unsuccessfully in 0:01:31
2019-10-03T22:53:15.2900159Z == clock drift check ==
2019-10-03T22:53:15.2928628Z   local time: Thu Oct  3 22:53:15 UTC 2019
2019-10-03T22:53:15.4439225Z   network time: Thu, 03 Oct 2019 22:53:15 GMT
2019-10-03T22:53:15.4439311Z == end clock drift check ==
2019-10-03T22:53:16.8120859Z ##[error]Bash exited with code '1'.
2019-10-03T22:53:16.8151552Z ##[section]Starting: Checkout
2019-10-03T22:53:16.8154038Z ==============================================================================
2019-10-03T22:53:16.8154095Z Task         : Get sources
2019-10-03T22:53:16.8154160Z Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

Rantanen · 2019-10-04T06:56:15Z

Fixed couple of naming issues and one spaces-within-parens violation. I think those were all of them. As far as I'm concerned, I think the code could be good to go. The compiler might disagree, my test run is still in progress - wanted to get the style issues pushed early so they wouldn't distract too much in the review.

src/libsyntax_pos/lib.rs

src/libsyntax/json.rs

src/libsyntax/json/tests.rs

rust-highfive · 2019-10-04T08:34:42Z

The job x86_64-gnu-llvm-6.0 of your PR failed (pretty log, raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

2019-10-04T06:51:57.7474491Z ##[command]git remote add origin https://github.com/rust-lang/rust
2019-10-04T06:51:57.7735648Z ##[command]git config gc.auto 0
2019-10-04T06:51:57.7831170Z ##[command]git config --get-all http.https://github.com/rust-lang/rust.extraheader
2019-10-04T06:51:57.7961166Z ##[command]git config --get-all http.proxy
2019-10-04T06:51:57.8072454Z ##[command]git -c http.extraheader="AUTHORIZATION: basic ***" fetch --force --tags --prune --progress --no-recurse-submodules --depth=2 origin +refs/heads/*:refs/remotes/origin/* +refs/pull/65074/merge:refs/remotes/pull/65074/merge
---
2019-10-04T07:55:29.9711995Z .................................................................................................... 1500/9103
2019-10-04T07:55:37.3540834Z .................................................................................................... 1600/9103
2019-10-04T07:55:47.0156763Z .................................................................................................... 1700/9103
2019-10-04T07:55:56.9295896Z .......i...............i............................................................................ 1800/9103
2019-10-04T07:56:04.5117020Z ..................................................................................................ii 1900/9103
2019-10-04T07:56:21.5725080Z iii................................................................................................. 2000/9103
2019-10-04T07:56:30.7917861Z .................................................................................................... 2200/9103
2019-10-04T07:56:33.5712899Z .................................................................................................... 2300/9103
2019-10-04T07:56:40.1274287Z .................................................................................................... 2400/9103
2019-10-04T07:56:45.9673838Z .................................................................................................... 2500/9103
---
2019-10-04T07:59:47.1056504Z .......................................................................................i............ 4700/9103
2019-10-04T07:59:55.3186074Z ...i................................................................................................ 4800/9103
2019-10-04T08:00:06.8267049Z .................................................................................................... 4900/9103
2019-10-04T08:00:12.1970339Z .................................................................................................... 5000/9103
2019-10-04T08:00:24.6165656Z ................................................................................ii.ii............... 5100/9103
2019-10-04T08:00:34.5118217Z .................................................................................................... 5300/9103
2019-10-04T08:00:44.9218881Z .................................................................................................... 5400/9103
2019-10-04T08:00:52.0572405Z ..............................................i..................................................... 5500/9103
2019-10-04T08:00:59.2961273Z .................................................................................................... 5600/9103
2019-10-04T08:00:59.2961273Z .................................................................................................... 5600/9103
2019-10-04T08:01:10.3684320Z .................................................................................................... 5700/9103
2019-10-04T08:01:18.4408987Z ...........................................ii...i..ii...........i................................... 5800/9103
2019-10-04T08:01:45.4940074Z .................................................................................................... 6000/9103
2019-10-04T08:01:55.3044672Z .................................................................................................... 6100/9103
2019-10-04T08:01:55.3044672Z .................................................................................................... 6100/9103
2019-10-04T08:02:08.2088696Z ................................................i..ii............................................... 6200/9103
2019-10-04T08:02:32.1328817Z .................................................................................................... 6400/9103
2019-10-04T08:02:34.5183128Z ........i........................................................................................... 6500/9103
2019-10-04T08:02:36.9351961Z ................................................................................i................... 6600/9103
2019-10-04T08:02:39.8428787Z .................................................................................................... 6700/9103
---
2019-10-04T08:07:27.2366257Z  finished in 5.731
2019-10-04T08:07:27.2558027Z Check compiletest suite=codegen mode=codegen (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
2019-10-04T08:07:27.4521991Z 
2019-10-04T08:07:27.4523145Z running 150 tests
2019-10-04T08:07:30.9910521Z i....iii......iii..iiii....i.............................i..i..................i....i.........ii.i.i 100/150
2019-10-04T08:07:33.0854403Z ..iiii..............i.........iii.i.......ii......
2019-10-04T08:07:33.0855850Z 
2019-10-04T08:07:33.0859806Z  finished in 5.830
2019-10-04T08:07:33.1056059Z Check compiletest suite=codegen-units mode=codegen-units (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
2019-10-04T08:07:33.2744555Z 
---
2019-10-04T08:07:35.5037113Z  finished in 2.397
2019-10-04T08:07:35.5237609Z Check compiletest suite=assembly mode=assembly (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
2019-10-04T08:07:35.7019241Z 
2019-10-04T08:07:35.7020267Z running 9 tests
2019-10-04T08:07:35.7021317Z iiiiiiiii
2019-10-04T08:07:35.7022121Z 
2019-10-04T08:07:35.7026304Z  finished in 0.179
2019-10-04T08:07:35.7249225Z Check compiletest suite=incremental mode=incremental (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
2019-10-04T08:07:35.9272433Z 
---
2019-10-04T08:07:55.4723657Z  finished in 19.747
2019-10-04T08:07:55.4963581Z Check compiletest suite=debuginfo mode=debuginfo-gdb+lldb (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
2019-10-04T08:07:55.6914440Z 
2019-10-04T08:07:55.6914808Z running 123 tests
2019-10-04T08:08:21.5127886Z .iiiii...i.....i..i...i..i.i.i..i.ii..i.i.....i..i....ii..........iiii..........i...ii...i.......ii. 100/123
2019-10-04T08:08:26.5512621Z i.i.i......iii.i.....ii
2019-10-04T08:08:26.5514371Z 
2019-10-04T08:08:26.5519688Z  finished in 31.055
2019-10-04T08:08:26.5530350Z Uplifting stage1 rustc (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
2019-10-04T08:08:26.5531001Z Copying stage2 rustc from stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
---
2019-10-04T08:21:47.9998121Z 
2019-10-04T08:21:47.9999033Z    Doc-tests core
2019-10-04T08:21:53.3189640Z 
2019-10-04T08:21:53.3190585Z running 2405 tests
2019-10-04T08:22:05.5716228Z ......iiiii......................................................................................... 100/2405
2019-10-04T08:22:17.0647393Z ...............................................................................ii................... 200/2405
2019-10-04T08:22:44.2507114Z .i.................................................................................................. 400/2405
2019-10-04T08:22:44.2507114Z .i.................................................................................................. 400/2405
2019-10-04T08:22:55.5572755Z ................................................i..i.................iiii........................... 500/2405
2019-10-04T08:23:17.3158426Z .................................................................................................... 700/2405
2019-10-04T08:23:28.5415172Z .................................................................................................... 800/2405
2019-10-04T08:23:39.7730405Z .................................................................................................... 900/2405
2019-10-04T08:23:50.8558964Z .................................................................................................... 1000/2405
---
2019-10-04T08:28:18.0756337Z 
2019-10-04T08:28:18.0756603Z running 992 tests
2019-10-04T08:28:40.4411697Z i................................................................................................... 100/992
2019-10-04T08:28:52.7406534Z .................................................................................................... 200/992
2019-10-04T08:29:01.6175689Z .................iii......i......i...i......i....................................................... 300/992
2019-10-04T08:29:07.8043904Z .................................................................................................... 400/992
2019-10-04T08:29:16.2149074Z ...................................i..i.................................ii.......................... 500/992
2019-10-04T08:29:32.1011131Z .................................................................................................... 700/992
2019-10-04T08:29:32.1011131Z .................................................................................................... 700/992
2019-10-04T08:29:40.8489504Z ..................iiii.............................................................................. 800/992
2019-10-04T08:29:56.7336797Z .................................................................................................... 900/992
2019-10-04T08:30:04.8305995Z ........................................iiii................................................
2019-10-04T08:30:04.8307227Z 
2019-10-04T08:30:04.8368208Z  finished in 203.177
2019-10-04T08:30:04.8387980Z Testing term stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
2019-10-04T08:30:05.0836200Z    Compiling term v0.0.0 (/checkout/src/libterm)
---
2019-10-04T08:34:34.9413781Z 
2019-10-04T08:34:34.9445779Z  finished in 4.017
2019-10-04T08:34:34.9467113Z Testing syntax_pos stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
2019-10-04T08:34:35.1427045Z    Compiling syntax_pos v0.0.0 (/checkout/src/libsyntax_pos)
2019-10-04T08:34:36.2251932Z error[E0599]: no method named `map` found for type `std::vec::Vec<NormalizedPos>` in the current scope
2019-10-04T08:34:36.2255508Z   --> src/libsyntax_pos/tests.rs:26:58
2019-10-04T08:34:36.2261155Z    |
2019-10-04T08:34:36.2261608Z 26 |         let actual_positions : Vec<_> = actual_positions.map(|nc| nc.pos).collect();
2019-10-04T08:34:36.2262342Z    |                                                          ^^^ method not found in `std::vec::Vec<NormalizedPos>`
2019-10-04T08:34:36.2262906Z    = note: the method `map` exists but the following trait bounds were not satisfied:
2019-10-04T08:34:36.2262906Z    = note: the method `map` exists but the following trait bounds were not satisfied:
2019-10-04T08:34:36.2263193Z            `&mut [NormalizedPos] : std::iter::Iterator`
2019-10-04T08:34:36.2263549Z            `&mut std::vec::Vec<NormalizedPos> : std::iter::Iterator`
2019-10-04T08:34:36.2430227Z error: aborting due to previous error
2019-10-04T08:34:36.2430301Z 
2019-10-04T08:34:36.2430622Z For more information about this error, try `rustc --explain E0599`.
2019-10-04T08:34:36.2693594Z error: could not compile `syntax_pos`.
---
2019-10-04T08:34:36.2786339Z == clock drift check ==
2019-10-04T08:34:36.2804301Z   local time: Fri Oct  4 08:34:36 UTC 2019
2019-10-04T08:34:36.5777186Z   network time: Fri, 04 Oct 2019 08:34:36 GMT
2019-10-04T08:34:36.5778187Z == end clock drift check ==
2019-10-04T08:34:37.2317521Z ##[error]Bash exited with code '1'.
2019-10-04T08:34:37.2375423Z ##[section]Starting: Checkout
2019-10-04T08:34:37.2377485Z ==============================================================================
2019-10-04T08:34:37.2377550Z Task         : Get sources
2019-10-04T08:34:37.2377619Z Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

petrochenkov · 2019-10-04T15:32:54Z

r? @matklad

Rantanen · 2019-10-04T16:42:05Z

I'll also try to remember run the libsyntax_pos unit tests this time. Unfortunately UDP tests are failing on my machine so running the full test suite is not possible for me. :|

rust-highfive · 2019-10-04T18:16:04Z

The job x86_64-gnu-llvm-6.0 of your PR failed (pretty log, raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

2019-10-04T17:56:28.8054148Z ##[command]git remote add origin https://github.com/rust-lang/rust
2019-10-04T17:56:28.8312524Z ##[command]git config gc.auto 0
2019-10-04T17:56:28.8400880Z ##[command]git config --get-all http.https://github.com/rust-lang/rust.extraheader
2019-10-04T17:56:28.8465965Z ##[command]git config --get-all http.proxy
2019-10-04T17:56:28.8618800Z ##[command]git -c http.extraheader="AUTHORIZATION: basic ***" fetch --force --tags --prune --progress --no-recurse-submodules --depth=2 origin +refs/heads/*:refs/remotes/origin/* +refs/pull/65074/merge:refs/remotes/pull/65074/merge
---
2019-10-04T18:03:47.7157516Z    Compiling serde_json v1.0.40
2019-10-04T18:03:50.5272471Z    Compiling tidy v0.1.0 (/checkout/src/tools/tidy)
2019-10-04T18:04:01.2609780Z     Finished release [optimized] target(s) in 1m 33s
2019-10-04T18:04:01.2696648Z tidy check
2019-10-04T18:04:01.5555780Z tidy error: /checkout/src/test/ui/json-bom-plus-crlf.rs:1: CR character
2019-10-04T18:04:01.5556628Z tidy error: /checkout/src/test/ui/json-bom-plus-crlf.rs:2: CR character
2019-10-04T18:04:01.5557125Z tidy error: /checkout/src/test/ui/json-bom-plus-crlf.rs:3: CR character
2019-10-04T18:04:01.5557564Z tidy error: /checkout/src/test/ui/json-bom-plus-crlf.rs:4: CR character
2019-10-04T18:04:01.5558038Z tidy error: /checkout/src/test/ui/json-bom-plus-crlf.rs:5: CR character
2019-10-04T18:04:01.5558454Z tidy error: /checkout/src/test/ui/json-bom-plus-crlf.rs:6: CR character
2019-10-04T18:04:01.5558887Z tidy error: /checkout/src/test/ui/json-bom-plus-crlf.rs:7: CR character
2019-10-04T18:04:01.5559310Z tidy error: /checkout/src/test/ui/json-bom-plus-crlf.rs:8: CR character
2019-10-04T18:04:01.5559720Z tidy error: /checkout/src/test/ui/json-bom-plus-crlf.rs:9: CR character
2019-10-04T18:04:01.5560301Z tidy error: /checkout/src/test/ui/json-bom-plus-crlf.rs:10: CR character
2019-10-04T18:04:01.5560712Z tidy error: /checkout/src/test/ui/json-bom-plus-crlf.rs:11: CR character
2019-10-04T18:04:01.5561146Z tidy error: /checkout/src/test/ui/json-bom-plus-crlf.rs:12: CR character
2019-10-04T18:04:01.5561808Z tidy error: /checkout/src/test/ui/json-bom-plus-crlf.rs:13: CR character
2019-10-04T18:04:01.5562305Z tidy error: /checkout/src/test/ui/json-bom-plus-crlf.rs:14: CR character
2019-10-04T18:04:01.5562940Z tidy error: /checkout/src/test/ui/json-bom-plus-crlf.rs:15: CR character
2019-10-04T18:04:01.5563358Z tidy error: /checkout/src/test/ui/json-bom-plus-crlf.rs:16: CR character
2019-10-04T18:04:02.3036707Z tidy error: /checkout/src/libsyntax/json/tests.rs: too many trailing newlines (2)
2019-10-04T18:04:03.5156055Z some tidy checks failed
2019-10-04T18:04:03.5156226Z 
2019-10-04T18:04:03.5156226Z 
2019-10-04T18:04:03.5157170Z command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/tidy" "/checkout/src" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "--no-vendor"
2019-10-04T18:04:03.5157284Z 
2019-10-04T18:04:03.5157312Z 
2019-10-04T18:04:03.5162382Z failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test src/tools/tidy
2019-10-04T18:04:03.5162484Z Build completed unsuccessfully in 0:01:36
2019-10-04T18:04:03.5162484Z Build completed unsuccessfully in 0:01:36
2019-10-04T18:04:03.5217091Z == clock drift check ==
2019-10-04T18:04:03.5246201Z   local time: Fri Oct  4 18:04:03 UTC 2019
2019-10-04T18:04:03.8019457Z   network time: Fri, 04 Oct 2019 18:04:03 GMT
2019-10-04T18:04:03.8020054Z == end clock drift check ==
2019-10-04T18:04:05.1954794Z ##[error]Bash exited with code '1'.
2019-10-04T18:04:05.1990377Z ##[section]Starting: Checkout
2019-10-04T18:04:05.1992408Z ==============================================================================
2019-10-04T18:04:05.1992481Z Task         : Get sources
2019-10-04T18:04:05.1992528Z Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

Rantanen · 2019-10-04T19:44:29Z

Also found a way to run the tidy tests. All the relevant tests passed on my machine, so I think this is good to go now from my end as long as I didn't screw things over in rebase and it's okay to leave the with_default_globals unit tests in for now.

matklad

LGTM, just a couple of nits left

src/libsyntax/json/tests.rs

src/librustc/ich/impls_syntax.rs

matklad · 2019-10-08T06:06:10Z

@bors r+

Thanks!

bors · 2019-10-08T06:06:11Z

📌 Commit bbf262d has been approved by matklad

@matklad

Fix the start/end byte positions in the compiler JSON output Track the changes made during normalization in the `SourceFile` and use this information to correct the `start_byte` and `end_byte` fields in the JSON output. This should ensure the start/end byte fields can be used to index the original file, even if Rust normalized the source code for parsing purposes. Both CRLF to LF and BOM removal are handled with this one. The rough plan was discussed with @matklad in rust-lang/rustfix#176 - although I ended up going with `u32` offset tracking so I wouldn't need to deal with `u32 + i32` arithmetics when applying the offset to the span byte positions. Fixes rust-lang#65029

@matklad

Fix the start/end byte positions in the compiler JSON output Track the changes made during normalization in the `SourceFile` and use this information to correct the `start_byte` and `end_byte` fields in the JSON output. This should ensure the start/end byte fields can be used to index the original file, even if Rust normalized the source code for parsing purposes. Both CRLF to LF and BOM removal are handled with this one. The rough plan was discussed with @matklad in rust-lang/rustfix#176 - although I ended up going with `u32` offset tracking so I wouldn't need to deal with `u32 + i32` arithmetics when applying the offset to the span byte positions. Fixes rust-lang#65029

Centril · 2019-10-08T13:44:03Z

Failed in #65199 (comment), @bors r-

…, r=petrochenkov" This reverts commit ef1ecbe, reversing changes made to fc8765d. That changed unfortunately broke rustfix on windows: rust-lang/rustfix#176 Specifically, what ef1ecbe did was to enforce normalization of \r\n to \n at file loading time, similarly to how we deal with Byte Order Mark. Normalization changes raw offsets in files, which are exposed via `--error-format=json`, and used by rusfix. The proper solution here (which also handles the latent case with BOM) is rust-lang#65074 However, since it's somewhat involved, and we are time sensitive, we prefer to revert the original change on beta.

matklad · 2019-10-10T15:23:16Z

Submitted #65273 which should relieve time pressure here, and increase our confidence that the backport is correct.

joelpalmer · 2019-10-21T11:44:12Z

Ping from Triage: Hi @Rantanen, any updates?

Rantanen · 2019-10-21T16:30:28Z

Oh, sorry. I was under the impression this was good to go and people were just busy with other stuff that the review was taking a while.

Let's squash the commits so that git history is not confusing about relative/absolute offsets?

I squashed the "Fix implementation" commits into the "Fix the start/end byte positions in the compiler JSON output", but still left the logically separate "Add unit tests" commits alone. Didn't know if the wish was to squash them all into one.

I've now done so. Do let me know if there's still something missing.

matklad · 2019-10-22T13:19:30Z

Sorry @Rantanen , I've missed force push, changes look good to me now!

@bors r+

bors · 2019-10-22T13:19:41Z

📌 Commit ff1860a has been approved by matklad

bors · 2019-10-22T19:09:59Z

⌛ Testing commit ff1860a with merge 93193b822279b7c5a943dc56de5e4296934a9ea0...

rust-highfive · 2019-10-22T19:13:33Z

Your PR failed (pretty log, raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

2019-10-22T19:11:29.5676683Z do so (now or later) by using -b with the checkout command again. Example:
2019-10-22T19:11:29.5676754Z 
2019-10-22T19:11:29.5676834Z   git checkout -b <new-branch-name>
2019-10-22T19:11:29.5676890Z 
2019-10-22T19:11:29.5677015Z HEAD is now at 93193b822 Auto merge of #65074 - Rantanen:json-byte-pos, r=matklad
2019-10-22T19:11:29.6094268Z ##[section]Starting: Collect CPU-usage statistics in the background
2019-10-22T19:11:29.6228055Z ==============================================================================
2019-10-22T19:11:29.6228152Z Task         : Bash
2019-10-22T19:11:29.6228245Z Description  : Run a Bash script on macOS, Linux, or Windows
---
2019-10-22T19:11:30.6842714Z ==============================================================================
2019-10-22T19:11:32.4133421Z Generating script.
2019-10-22T19:11:32.4196578Z ========================== Starting Command Output ===========================
2019-10-22T19:11:32.4579240Z ##[command]"C:\windows\system32\cmd.exe" /D /E:ON /V:OFF /S /C "CALL "D:\a\_temp\98eab51a-8581-4c15-a65d-9cca954d7520.cmd""
2019-10-22T19:11:44.8694315Z iwr : The remote name could not be resolved: 'rust-lang-ci-mirrors.s3-us-west-1.amazonaws.com'
2019-10-22T19:11:44.8694644Z At line:1 char:43
2019-10-22T19:11:44.8694744Z + ... yContinue'; iwr -outf sccache\sccache.exe https://rust-lang-ci-mirror ...
2019-10-22T19:11:44.8695434Z +                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2019-10-22T19:11:44.8696092Z     + CategoryInfo          : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-WebRequest], WebExc 
2019-10-22T19:11:44.8696701Z    eption
2019-10-22T19:11:44.8697698Z     + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand
2019-10-22T19:11:44.8697978Z  
2019-10-22T19:11:45.0008378Z ##[error]Cmd.exe exited with code '1'.
2019-10-22T19:11:45.0794337Z ##[section]Starting: Upload CPU usage statistics
2019-10-22T19:11:45.0988325Z ==============================================================================
2019-10-22T19:11:45.0988464Z Task         : Bash
2019-10-22T19:11:45.0988546Z Description  : Run a Bash script on macOS, Linux, or Windows
---
2019-10-22T19:11:45.4330899Z ========================== Starting Command Output ===========================
2019-10-22T19:11:45.4337286Z [command]"C:\Program Files\Git\bin\bash.exe" --noprofile --norc /d/a/_temp/e72207d0-9cf0-4fcd-8292-6d34481d4715.sh
2019-10-22T19:11:45.4830315Z /d/a/_temp/e72207d0-9cf0-4fcd-8292-6d34481d4715.sh: line 1: aws: command not found
2019-10-22T19:11:45.4877604Z 
2019-10-22T19:11:45.4900118Z ##[error]Bash exited with code '127'.
2019-10-22T19:11:45.4977285Z ##[section]Starting: Checkout
2019-10-22T19:11:45.5089986Z ==============================================================================
2019-10-22T19:11:45.5090088Z Task         : Get sources
2019-10-22T19:11:45.5090198Z Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

bors · 2019-10-22T19:13:45Z

💔 Test failed - checks-azure

Rantanen · 2019-10-22T19:41:07Z

iwr : The remote name could not be resolved: 'rust-lang-ci-mirrors.s3-us-west-1.amazonaws.com'

I'm assuming this is a transient error. If it isn't and I need to do something about it, I'll need a bit more information than that. :)

matklad · 2019-10-23T06:31:24Z

hm, indeed seems spurious

@bors retry

@matklad

Fix the start/end byte positions in the compiler JSON output Track the changes made during normalization in the `SourceFile` and use this information to correct the `start_byte` and `end_byte` fields in the JSON output. This should ensure the start/end byte fields can be used to index the original file, even if Rust normalized the source code for parsing purposes. Both CRLF to LF and BOM removal are handled with this one. The rough plan was discussed with @matklad in rust-lang/rustfix#176 - although I ended up going with `u32` offset tracking so I wouldn't need to deal with `u32 + i32` arithmetics when applying the offset to the span byte positions. Fixes rust-lang#65029

@matklad

Fix the start/end byte positions in the compiler JSON output Track the changes made during normalization in the `SourceFile` and use this information to correct the `start_byte` and `end_byte` fields in the JSON output. This should ensure the start/end byte fields can be used to index the original file, even if Rust normalized the source code for parsing purposes. Both CRLF to LF and BOM removal are handled with this one. The rough plan was discussed with @matklad in rust-lang/rustfix#176 - although I ended up going with `u32` offset tracking so I wouldn't need to deal with `u32 + i32` arithmetics when applying the offset to the span byte positions. Fixes rust-lang#65029

@ghost

Rollup of 9 pull requests Successful merges: - #64639 (Stabilize `#[non_exhaustive]` (RFC 2008)) - #65074 (Fix the start/end byte positions in the compiler JSON output) - #65315 (Intern place projection) - #65685 (Fix check of `statx` and handle EPERM) - #65731 (Prevent unnecessary allocation in PathBuf::set_extension.) - #65740 (Fix default "disable-shortcuts" feature value) - #65787 (move panictry! to where it is used.) - #65789 (move Attribute::with_desugared_doc to librustdoc) - #65790 (move report_invalid_macro_expansion_item to item.rs) Failed merges: r? @ghost

…, r=petrochenkov" This reverts commit ef1ecbe, reversing changes made to fc8765d. That changed unfortunately broke rustfix on windows: rust-lang/rustfix#176 Specifically, what ef1ecbe did was to enforce normalization of \r\n to \n at file loading time, similarly to how we deal with Byte Order Mark. Normalization changes raw offsets in files, which are exposed via `--error-format=json`, and used by rusfix. The proper solution here (which also handles the latent case with BOM) is rust-lang#65074 However, since it's somewhat involved, and we are time sensitive, we prefer to revert the original change on beta.

rust-highfive assigned petrochenkov Oct 3, 2019

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Oct 3, 2019

Rantanen force-pushed the json-byte-pos branch 2 times, most recently from 994bda0 to d0dc2a8 Compare October 4, 2019 06:51

matklad reviewed Oct 4, 2019

View reviewed changes

src/libsyntax_pos/lib.rs Outdated Show resolved Hide resolved

src/libsyntax_pos/lib.rs Outdated Show resolved Hide resolved

src/libsyntax/json.rs Outdated Show resolved Hide resolved

src/libsyntax/json/tests.rs Show resolved Hide resolved

rust-highfive assigned matklad and unassigned petrochenkov Oct 4, 2019

Rantanen force-pushed the json-byte-pos branch from d0dc2a8 to c80d54b Compare October 4, 2019 17:55

Rantanen force-pushed the json-byte-pos branch from c80d54b to e492ca8 Compare October 4, 2019 19:38

matklad reviewed Oct 5, 2019

View reviewed changes

src/libsyntax/json/tests.rs Outdated Show resolved Hide resolved

src/libsyntax/json/tests.rs Outdated Show resolved Hide resolved

src/libsyntax/json/tests.rs Show resolved Hide resolved

src/librustc/ich/impls_syntax.rs Outdated Show resolved Hide resolved

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Oct 8, 2019

matklad added the beta-nominated Nominated for backporting to the compiler in the beta channel. label Oct 8, 2019

Centril mentioned this pull request Oct 8, 2019

Rollup of 7 pull requests #65199

Closed

Centril mentioned this pull request Oct 8, 2019

Rollup of 5 pull requests #65203

Closed

Fix the start/end byte positions in the compiler JSON output

ff1860a

Rantanen force-pushed the json-byte-pos branch from 89318cf to ff1860a Compare October 21, 2019 16:29

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Oct 22, 2019

bors added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Oct 22, 2019

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Oct 23, 2019

Centril mentioned this pull request Oct 25, 2019

Rollup of 5 pull requests #65797

Closed

Centril mentioned this pull request Oct 25, 2019

Rollup of 9 pull requests #65804

Merged

bors merged commit ff1860a into rust-lang:master Oct 25, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix the start/end byte positions in the compiler JSON output #65074

Fix the start/end byte positions in the compiler JSON output #65074

Rantanen commented Oct 3, 2019

rust-highfive commented Oct 3, 2019

Rantanen commented Oct 3, 2019 •

edited

Loading

rust-highfive commented Oct 3, 2019

Rantanen commented Oct 4, 2019

rust-highfive commented Oct 4, 2019

petrochenkov commented Oct 4, 2019

Rantanen commented Oct 4, 2019

rust-highfive commented Oct 4, 2019

Rantanen commented Oct 4, 2019

matklad left a comment

matklad commented Oct 8, 2019

bors commented Oct 8, 2019

Centril commented Oct 8, 2019

matklad commented Oct 10, 2019

joelpalmer commented Oct 21, 2019

Rantanen commented Oct 21, 2019

matklad commented Oct 22, 2019 •

edited

Loading

bors commented Oct 22, 2019

bors commented Oct 22, 2019

rust-highfive commented Oct 22, 2019

bors commented Oct 22, 2019

Rantanen commented Oct 22, 2019

matklad commented Oct 23, 2019

Fix the start/end byte positions in the compiler JSON output #65074

Fix the start/end byte positions in the compiler JSON output #65074

Conversation

Rantanen commented Oct 3, 2019

rust-highfive commented Oct 3, 2019

Rantanen commented Oct 3, 2019 • edited Loading

rust-highfive commented Oct 3, 2019

Rantanen commented Oct 4, 2019

rust-highfive commented Oct 4, 2019

petrochenkov commented Oct 4, 2019

Rantanen commented Oct 4, 2019

rust-highfive commented Oct 4, 2019

Rantanen commented Oct 4, 2019

matklad left a comment

Choose a reason for hiding this comment

matklad commented Oct 8, 2019

bors commented Oct 8, 2019

Centril commented Oct 8, 2019

matklad commented Oct 10, 2019

joelpalmer commented Oct 21, 2019

Rantanen commented Oct 21, 2019

matklad commented Oct 22, 2019 • edited Loading

bors commented Oct 22, 2019

bors commented Oct 22, 2019

rust-highfive commented Oct 22, 2019

bors commented Oct 22, 2019

Rantanen commented Oct 22, 2019

matklad commented Oct 23, 2019

Rantanen commented Oct 3, 2019 •

edited

Loading

matklad commented Oct 22, 2019 •

edited

Loading