Catch possible tokenizer panics

[Xanewok]

When using new_parser_from_source_str sometimes it can panic on incomplete input (e.g. bare }), so here we use the Result-returning maybe_new_parser_from_source_str variant to catch possible errors and diagnostics to emit later.

Closes #3239

related #753
cc @topecongiro

[Xanewok]

Changed this to return report here for parse errors, otherwise we eagerly ?-return an Err here:

rustfmt/src/formatting.rs

Line 76 in 66c98db

let krate = parse_crate(input, &parse_session, config, &mut report)?;

but we only add errors to Session if we have Ok(FormatReport)

rustfmt/src/formatting.rs

Lines 49 to 57 in 66c98db

	format_result.map(|report| {
	{
	let new_errors = &report.internal.borrow().1;
	self.errors.add(new_errors);
	}
	report
	})
	})

I changed this because I expected session.has_parsing_errors() call to work:
https://github.com/rust-lang/rustfmt/pull/3240/files#diff-382d140f826c9bd99f62116b9712fad3R299
and it seemed like a special case that it didn't.

However, for IO errors we still will get Err(_) from Session::format(...) and session.has_operational_errors() will return false (should we also change that?).

[topecongiro]

However, for IO errors we still will get Err(_) from Session::format(...) and session.has_operational_errors() will return false (should we also change that?).

I think so, but that can be done in a separate PR. I would rather fix this via refactoring the whole error handling around Session, as currently it is error prone.

[Xanewok]

This should be ready to review now

[topecongiro]

Thank you!

[wada314]

Hello,

I'm running "$ cargo test stdin_parser_panic_caught" in nightly Rust, Windows 10.
The test itself successfully finishes, but the panic!() outputs are not suppressed and displayed in my console.
Is this an expected behavior? Or is this something happening only in my environment?

thanks,

running 1 test
error: this file contains an un-closed delimiter
 --> <stdin>:1:2
  |
1 | {
  | -^
  | |
  | un-closed delimiter

error: expected item, found `{`
 --> <stdin>:1:1
  |
1 | {
  | ^ expected item

error: unexpected close delimiter: `}`
 --> <stdin>:1:1
  |
1 | }
  | ^ unexpected close delimiter

test test::stdin_parser_panic_caught ... ok