Data source name format

[abhineet13]

Background [Optional]

Hi, the data source tab lists data source name as the last string in storage path, for example

spark.read.parquet("s3://bucket/folder/partition=2020-01-01") 

will have a data source name of "partition=2020-01-01" and

val df= spark.read.parquet("s3://bucket/folder")
df.createOrReplaceTempView("df")
spark.sql("select * from df where partition=2020-01-01")

will create a data source name of "folder" in Spline UI.

Question

Is it possible to keep data source naming consistent at folder level, otherwise for daily jobs Spline UI will show multiple data source names for each partition.

Thanks

[wajda]

Unfortunately, it all depends on what the agent provides. The UI representation is as good as the metadata that Spline server receives.
In your example Spark actually sees two different datasource URIs according to what is provided to the Read operation. The UI simply shown the last portion of the URI as a short name as it simply doesn't have any more precise information.
We expect this problem to be at least partially solved with the help of Data source management feature that we plan to implement in the future (see #689)

[abhineet13]

ok, got it, for now we will modify the jobs accordingly, thanks for responding.