coprocessor: parsing error is never returned

[BusyJay]

When it fails to parse a request, the error is wrapped in an dummy handler which is returned from the function with a fake context and supposed to be executed inside read pool.

tikv/src/coprocessor/endpoint.rs

Lines 196 to 211 in e53a8c2

	match self.try_parse_request(req, peer, is_streaming) {
	Ok(v) => v,
	Err(err) => {
	// If there are errors when parsing requests, create a dummy request handler.
	let builder =
	box |_, _: &_| Ok(cop_util::ErrorRequestHandler::new(err).into_boxed());
	let req_ctx = ReqContext::new(
	"invalid",
	kvrpcpb::Context::new(),
	&[],
	Duration::from_secs(60), // Large enough to avoid becoming outdated error
	None,
	None,
	None,
	);
	(builder, req_ctx)

However endpoint will continue to get a snapshot first using the fake context, which will always fail as it's an invalid context.

tikv/src/coprocessor/endpoint.rs

Lines 344 to 357 in e53a8c2

	future::result(tracker.req_ctx.deadline.check_if_exceeded())
	.and_then(move |_| {
	Self::async_snapshot(engine, &tracker.req_ctx.context)
	.map(|snapshot| (tracker, snapshot))
	})
	.and_then(move |(tracker, snapshot)| {
	// When snapshot is retrieved, deadline may exceed.
	future::result(tracker.req_ctx.deadline.check_if_exceeded())
	.map(|_| (tracker, snapshot))
	})
	.and_then(move |(tracker, snapshot)| {
	future::result(handler_builder.call_box((snapshot, &tracker.req_ctx)))
	.map(|handler| (tracker, handler))
	});

So a strange error from raftstore is returned instead of the actual error which is the parsing error in the first place.

The bug is introduced in #3515, and affects v2.1.x and v3.0.x, needs to be fixed soon.

@disksing @breeswish PTAL

[siddontang]

Finally, we find this problem 😭

Btw, why parsing failed? the request is too big? @BusyJay

[disksing]

The request can be unmarshalled in Go. I guess the problem is the DAG request recurses too deep.

[breezewish]

Interesting finding. Looks like we must find a way that parsing failed errors are returned immediately instead of partially finished the pipe line.

[breezewish]

Fixed in #4303