Read hdfs file hit exception during the read process

[Nizifan]

Alluxio Version:
2.7

Describe the bug
When try to read ~2G file from hdfs through Alluxio, command hit IO error and log indicate Java exception is hit.
And it read out correct but partial result

[root@jssz-aiinf-ml-gpu-04 alluxio-2.7.2]# cp /runtime-mnt/alluxio/ogv/ogv-hdfs/alluxio-fuse/ogv-hdfs/log_date=20220117/part-00000-ed2f54f4-90f1-4551-a05b-96345174ad5f.c000 ./temp.out
cp: error reading '/runtime-mnt/alluxio/ogv/ogv-hdfs/alluxio-fuse/ogv-hdfs/log_date=20220117/part-00000-ed2f54f4-90f1-4551-a05b-96345174ad5f.c000': Input/output error
cp: failed to extend './temp.out': Input/output error
[root@jssz-aiinf-ml-gpu-04 alluxio-2.7.2]# cp /runtime-mnt/alluxio/ogv/ogv-hdfs/alluxio-fuse/ogv-hdfs/log_date=20220120/part-00000-aaec8d8d-aa5b-4562-ba70-319e47380d64.c000 ./temp1.out
cp: error reading '/runtime-mnt/alluxio/ogv/ogv-hdfs/alluxio-fuse/ogv-hdfs/log_date=20220120/part-00000-aaec8d8d-aa5b-4562-ba70-319e47380d64.c000': Input/output error
cp: failed to extend './temp1.out': Input/output error
[root@jssz-aiinf-ml-gpu-04 alluxio-2.7.2]# ls -lh
-rw-r--r-- 1 root root 2.2M Jan 25 10:46 temp.out
-rw-r--r-- 1 root root 1.1G Jan 25 10:47 temp1.out

original files are around 2.4g
[root@jssz-aiinf-ml-gpu-04 alluxio-2.7.2]# ls -lh /runtime-mnt/alluxio/ogv/ogv-hdfs/alluxio-fuse/ogv-hdfs/log_date=20220120/part-00000-aaec8d8d-aa5b-4562-ba70-319e47380d64.c000
-rw-r--r-- 1 root root 2.4G Jan 21 06:45 /runtime-mnt/alluxio/ogv/ogv-hdfs/alluxio-fuse/ogv-hdfs/log_date=20220120/part-00000-aaec8d8d-aa5b-4562-ba70-319e47380d64.c000

Logs:
2022-01-25 08:54:49,852 ERROR AlluxioJniFuseFileSystem - Failed to read, path: /ogv-hdfs/log_date=20220117/part-00000-ed2f54f4-90f1-4551-a05b-96345174ad5f.c000 size: 131072 offset: 2228224
java.lang.IllegalArgumentException
at java.nio.Buffer.position(Buffer.java:244)
at alluxio.client.block.stream.SharedGrpcDataReader.readChunk(SharedGrpcDataReader.java:115)
at alluxio.client.block.stream.BlockInStream.readChunk(BlockInStream.java:489)
at alluxio.client.block.stream.BlockInStream.read(BlockInStream.java:316)
at alluxio.client.file.AlluxioFileInStream.read(AlluxioFileInStream.java:184)
at alluxio.fuse.AlluxioJniFuseFileSystem.readInternal(AlluxioJniFuseFileSystem.java:408)
at alluxio.fuse.AlluxioJniFuseFileSystem.lambda$read$5(AlluxioJniFuseFileSystem.java:383)
at alluxio.fuse.AlluxioFuseUtils.call(AlluxioFuseUtils.java:318)
at alluxio.fuse.AlluxioJniFuseFileSystem.read(AlluxioJniFuseFileSystem.java:383)
at alluxio.jnifuse.AbstractFuseFileSystem.readCallback(AbstractFuseFileSystem.java:180)

If use hdfs client, it can read out correct result correctly.

[LuQQiu]

@Nizifan why enabling alluxio.fuse.shared.caching.reader.enabled to use SharedGrpcDataReader?
@apc999 did we plan to remove SharedGrpcDataReader and what's the reason for us to plan to remove that class?

[apc999]

to @LuQQiu No concrete plan to remove SharedGrpcDataReader until we figure out more efficient way to reuse transport

to @Nizifan flag alluxio.fuse.shared.caching.reader.enabled is experimental, we do not recommend turning on this flag

[Nizifan]

Thanks @apc999 and @LuQQiu
Disabling this setting solve the problem, and it is not a 100% bug, we try several hdfs read but only 1 instance hit this issue but it is 100% repro in this case.

The config is copy-paste from fluid from fluid default. Do you suggest to turn it off in fluid?
setDefaultProperties(runtime, value, "alluxio.fuse.shared.caching.reader.enabled", "true")

[cheyang]

@apc999 @LuQQiu The properties are for the same blocks concurrent access from #12453 which is from Microsoft. Could you make double check?

[Nizifan]

If this config will greatly impact read performance should we dig a bit more on while shared reader will cause read failure?

[LuQQiu]

We will take a deeper look into the issue and see how to handle the multiple threads/processes read same big file issue

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in two weeks if no further activity occurs. Thank you for your contributions.