[Remote Store] Parallel segment downloads in IndexShard

[shourya035]

Is your feature request related to a problem? Please describe.
As of today, we are using the Directory#copyFrom method synchronously. The invocation is done on a simple for loop which might end up slowing down the shard recovery process.

OpenSearch/server/src/main/java/org/opensearch/index/shard/IndexShard.java

Lines 4567 to 4577 in 84dffac

	for (String file : uploadedSegments.keySet()) {
	long checksum = Long.parseLong(uploadedSegments.get(file).getChecksum());
	if (overrideLocal || localDirectoryContains(storeDirectory, file, checksum) == false) {
	if (localSegmentFiles.contains(file)) {
	storeDirectory.deleteFile(file);
	}
	storeDirectory.copyFrom(remoteDirectory, file, file, IOContext.DEFAULT);
	downloadedSegments.add(file);
	} else {
	skippedSegments.add(file);
	}

We should look into making this copyFrom invocation in parallel to speed up segment downloads during shard recovery

Describe the solution you'd like
Use a new threadpool called remote_download to download these segment files in parallel

[anasalkouz]

Seems related to #8187

[kotwanikunal]

Seems related to #8187

It's actually related to #8596