Save operational params in the same way with delta io #1054
Conversation
ismoshkov
requested review from
houqp,
xianwill,
wjones127,
fvaleye,
roeap,
rtyler and
mosyp
as code owners
rtyler
added
enhancement
| @@ -493,7 +493,7 @@ async fn test_commit_info() -> Result<(), Box<dyn Error>> { | |||
| assert_eq!(last_commit["readVersion"], json!(version)); | |||
| assert_eq!( | |||
| last_commit["operationParameters"]["targetSize"], | |||
| json!(2_000_000) | |||
| json!("2000000") | |||
There was a problem hiding this comment.
I'd also like to see a test for partitionValues. Could you add that?
There was a problem hiding this comment.
Sorry, could you please clarify your request? I ask because "partitionValues" is not part of the commit info and operating params that have been changed in this PR.
Perhaps you want to check that "partitionBy" as part of the operating params is built correctly?
If you want exactly "partitionValues", I can figure out how to check that. But it looks like it should be in a separate PR.
There was a problem hiding this comment.
Yes, sorry. I meant "partitionBy".
There was a problem hiding this comment.
Added separated unit test, because the most of commit info sections doesn't pass operation info. So they doesn't contains operational parameters which I can use to check.
* fix commit info building
# Description Currently writing "operationParameters" in commit info is misaligned with delta io connector. [Here](https://github.com/delta-io/delta/blob/36a7edb8cf507e713700ba827c5fb5ad32b9163e/core/src/main/scala/org/apache/spark/sql/delta/actions/actions.scala#L695) the sample of structure which is used in delta io. So the goal of this PR is to align with delta io approach and the PR do two thins: convert all values to string and delete keys with null values. # Related Issue(s) Closes [issue delta-io#1017](delta-io#1017) Co-authored-by: Ilya Moshkov <[email protected]>
Description
Currently writing "operationParameters" in commit info is misaligned with delta io connector.
Here the sample of structure which is used in delta io.
So the goal of this PR is to align with delta io approach and the PR do two thins: convert all values to string and delete keys with null values.
Related Issue(s)
Closes issue #1017