fix(MemTable): make it cancel-safe and fix parallelism

[DDtKey]

Which issue does this PR close?

The issue is close to #5178, but for MemTable.

Rationale for this change

1. It should be cancel-safe
2. There was an issue with join - await was sequential
3. There was possible to get a panic instead of Error

What changes are included in this PR?

Are these changes tested?

Existed tests are passing

Are there any user-facing changes?

[alamb]

Thanks @DDtKey ❤️

I left a suggestion that I think would make it even better but I think this is an improvement (internal error vs panic 👍 ) already and could go in as is

[alamb]

I think it would help if we could preserve the original error here rather than turning it into an internal error

Perhaps using DataFusionError::Context

Suggested change

	data.push(result.map_err(|_| {
	DataFusionError::Internal(
	"MemTable::load could not join task".to_string(),
	)
	data.push(result.map_err(|e| {
	DataFusionError::Context(
	"MemTable::load could not join task".to_string(), Box::new(e)
	)

[DDtKey]

Context requires DataFusionError, so it's anyway should be wrapped to something 🤔

Could be data.push(result.map_err(|e| DataFusionError::External(Box::new(e)))??), WDYT?

[alamb]

That would definitely be better than 👍

[DDtKey]

Done 🙂

[alamb]

Thanks again @DDtKey

[ursabot]

Benchmark runs are scheduled for baseline = 816a0f8 and contender = f63b972. f63b972 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ec2-t3-xlarge-us-east-2] ec2-t3-xlarge-us-east-2
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on test-mac-arm] test-mac-arm
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ursa-i9-9960x] ursa-i9-9960x
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ursa-thinkcentre-m75q] ursa-thinkcentre-m75q
Buildkite builds:
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java