Is RPC GetFileMetadata in Ballista still needed?

[yjshen]

While working on #932 (comment), I find this GetFileMetadata RPC is not really used, and the intention of the RPC is not clear to me. Do we still need it? Or it's deprecated, and we could safely remove it?

@andygrove, any ideas?

[andygrove]

The intent is to update BallistaContext::read_csv / read_parquet methods to retrieve file list and meta data from the scheduler rather than from the local file system so that we can use clients outside of the cluster. For example, I may have a cluster in AWS and I want to run the client on my laptop.

[andygrove]

See apache/datafusion-ballista#22

[houqp]

Ha, good to know. @rdettai FYI since this is related to the new query planning design you are working on.