More intelligent diffing in changes of file.serialize state #48609

anlutro · 2018-07-16T19:07:06Z

I wanted some feedback on this minor feature before I tidy it up, write tests etc. - I've confirmed it works with a local checkout of the salt codebase.

Basically I have a JSON config file being generated by a Go application, and I want to manage parts of it with Salt. However, because Python and Go order their JSON differently (order of keys mostly), the diff output is always massive, making it impossible to see what changed.

I found salt.utils.dictdiffer which does exactly what I want in this scenario. I added an optional argument to file.serializer which switches the behaviour of the state's changes output.

Some problems/questions which I'd like feedback on:

Should this behaviour be the default or is it best to keep it behind this optional argument? Is the name of the argument okay?
The changes output is potentially misleading, because even if the serialized data doesn't change, the file's contents (indentation or whatever) may change, which could screw up onchanges etc.
dictdiffer only works on dicts, but it's possible to serialize other data types such as lists.

DmitryKuzmenko · 2018-07-27T08:41:27Z

Basically I'm OK with the idea.

Should this behaviour be the default or is it best to keep it behind this optional argument?

False by default as for me, to not break this for others.

Is the name of the argument okay?

Maybe shorter? show_diff? But I like to hear an opinion of somebody else.

The changes output is potentially misleading, because even if the serialized data doesn't change, the file's contents (indentation or whatever) may change, which could screw up onchanges etc.

Maybe just document this point?

dictdiffer only works on dicts, but it's possible to serialize other data types such as lists.

Not sure it's a good idea but the simplest way is if it's not a dict make it a dict {'data': existing_data} or better implement a list diffing logic.

anlutro · 2018-07-28T14:17:46Z

Isn't show_diff reserved as a global argument meant to control whether any diff at all is shown? I think it sounds a little bit misleading anyway - if it's false, the diff is still shown, it's just shown in a different way.

DmitryKuzmenko · 2018-07-30T08:48:12Z

@anlutro I'm ok then. Sorry for misleading

anlutro · 2018-07-30T08:49:32Z

Cool. I will work on the list stuff as well as docs a bit later today, hopefully.

rallytime

I am OK with this change, but the current behavior should be the default. I'd also like @terminalmage to take a look here as well.

rallytime · 2018-07-30T14:02:04Z

salt/states/file.py

@@ -6374,6 +6375,7 @@ def serialize(name,
              encoding=None,
              encoding_errors='strict',
              serializer_opts=None,
+              diff_serializer_data=True,


This should be False by default to keep the previous behavior, yes?

terminalmage

I like anything that helps normalize changes. However, I don't think we need to hide this behavior behind an argument. I'm not sure I understand what changing the format of the diffs in the changes key of the return data is going to "break".

IMO this behavior should be the new default and we don't need to make it contingent on an argument passed to the state.

rallytime · 2018-07-31T14:55:12Z

OK, after talking to @terminalmage about this offline, I agree with him. We definitely need a mention of the change in the release notes though.

anlutro · 2018-07-31T16:07:21Z

Okay, cool. I just need to figure out how to do the diffing, then. I can't find anything in dictdiffer that does list diffs in particular, and the listdiffer module seems to be aimed at lists of dictionaries in particular. Will I need to implement it myself? What would the module name be, since listdiffer is already taken?

anlutro · 2018-07-31T16:50:11Z

On second thought, after considering how difficult smartly diffing lists is, maybe I should just not do that... But that might mean that data structures such as lists of dicts or lists of lists of dicts won't get smartly diffed.

Maybe I can simply wrap the dictdiff call in an if isinstance(data, dict) and call it a day?

terminalmage

@anlutro Can you elaborate on what you mean by list diffing? The recursive diffing in dictdiffer does show where lists differ.

To me this looks good as-is, but since I'm not sure where you're coming from, I want to make sure we're on the same page before I approve this PR for merging.

anlutro · 2018-08-20T15:35:22Z

I mean if we have this JSON data: [1,2,3] and we change it to [1,2,4] we could show that 3 has been removed and 4 has been added. But when I thought more about it, it would be really difficult to implement.

terminalmage · 2018-08-20T19:09:22Z

Oh OK, so if the data is being serialized and was formed as a list instead of a dict.

I guess you could always do type checking like you suggested, but we would still want there to be something in the changes key of the return dictionary if we were dealing with lists. the reason is that requisites such as onchanges look at the changes dict to detect whether or not any changes happened, so if we want these requisites to work properly, we do need to have some value there.

isbm

I have no much to add here what already has been said.

cachedout · 2018-08-28T18:03:30Z

@anlutro Are you ready for a final review here? I saw that you removed the WIP indicator from the PR title.

anlutro · 2018-08-28T18:06:54Z

Possibly - I wanted to see if any tests were failing first but seemed like Jenkins was freaking out.

…

On Tue, Aug 28, 2018, 20:03 Mike Place ***@***.***> wrote: @anlutro <https://github.com/anlutro> Are you ready for a final review here? I saw that you removed the WIP indicator from the PR title. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#48609 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAJ9Fb078ADucwNSNgJwkephJQ61jCWkks5uVYYFgaJpZM4VRo2O> .

rallytime · 2018-08-28T18:15:15Z

@anlutro I think this test failure is related: https://jenkinsci.saltstack.com/job/pr-kitchen-ubuntu1604-py2/job/PR-48609/10/

Can you take a look?

terminalmage

There is still nothing in ret['changes'] if the data is non-dictionary. We need to have at least something in there for onchanges requisites to work.

anlutro · 2018-08-29T20:29:43Z

@rallytime Ah - fixed that now. Let's see if other failures pop up.

@terminalmage ret['changes'] will be set by file.check_managed_changes or file.manage_file, we just overwrite it if we think we can.

Sorry for the late replies, I'm out travelling.

terminalmage · 2018-08-29T20:46:07Z

Ah ok, thanks!

rallytime · 2018-08-30T14:02:04Z

@anlutro It's still failing, unfortunately:

https://jenkinsci.saltstack.com/job/pr-kitchen-ubuntu1604-py3/job/PR-48609/13/

cachedout · 2018-10-16T11:14:19Z

@anlutro Any updates here?

anlutro · 2018-10-16T19:33:46Z

Hey, sorry for going quiet. I see there's a test failure but it doesn't make any sense to me. It looks like self.run_state is returning a boolean - why? I'm pretty sure none of the code I added/modified could trigger this behavior.

terminalmage

I've started new tests since the old results are aged out of Jenkins. Pending getting them fixed, I think this is good to merge.

cachedout · 2018-12-10T20:46:54Z

@anlutro It looks like that test run failed. Can you take a look please? https://jenkinsci.saltstack.com/job/pr-kitchen-centos7-py3/job/PR-48609/40/testReport/junit/integration.states.test_file/FileTest/test_serializer_deserializer_opts/

terminalmage · 2018-12-10T21:04:57Z

I just pushed a commit to this PR to fix the docs build issue, @anlutro. You'll need to pull before you make any more changes.

cachedout · 2018-12-10T21:46:20Z

I'll rebase this PR in the mean time.

terminalmage · 2018-12-11T13:48:33Z

I found and fixed a problem in this PR, we were including the dictdiffer object in the changes dict and msgpack couldn't serialize it. 1af036a should fix this.

rallytime requested review from a team July 26, 2018 21:37

rallytime reviewed Jul 30, 2018

View reviewed changes

rallytime requested a review from terminalmage July 30, 2018 14:03

terminalmage reviewed Jul 31, 2018

View reviewed changes

rallytime requested a review from terminalmage August 14, 2018 18:53

terminalmage reviewed Aug 20, 2018

View reviewed changes

isbm reviewed Aug 21, 2018

View reviewed changes

anlutro changed the title ~~[wip] implement file.serialize dict diffing~~ More intelligent diffing in changes of file.serialize state Aug 21, 2018

terminalmage suggested changes Aug 29, 2018

View reviewed changes

anlutro added 3 commits September 5, 2018 19:26

implement file.serialize dict diffing

260b8c1

existing_data must always be set

e4a0887

update changes.diff instead of changes

d00859e

Merge branch 'develop' into impl-file-serialize-diff-dict

c5a94bd

gtmanfred and others added 3 commits October 16, 2018 14:35

Merge branch 'develop' into impl-file-serialize-diff-dict

f674e99

Merge branch 'develop' into impl-file-serialize-diff-dict

14d3b2e

Merge branch 'develop' into impl-file-serialize-diff-dict

d5c63ec

terminalmage approved these changes Dec 10, 2018

View reviewed changes

Properly configure roster toctree and dir roster RST file

02c7093

Mike Place and others added 2 commits December 10, 2018 14:46

Merge branch 'develop' into impl-file-serialize-diff-dict

204822a

Return the diff dict instead of the dictdiffer object

1af036a

terminalmage merged commit 63a75b3 into saltstack:develop Dec 11, 2018

garethgreenaway mentioned this pull request Jul 28, 2020

[merge jam] Master port 48609 - file.serialize #58055

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

More intelligent diffing in changes of file.serialize state #48609

More intelligent diffing in changes of file.serialize state #48609

anlutro commented Jul 16, 2018

DmitryKuzmenko commented Jul 27, 2018

anlutro commented Jul 28, 2018

DmitryKuzmenko commented Jul 30, 2018

anlutro commented Jul 30, 2018

rallytime left a comment

rallytime Jul 30, 2018

terminalmage left a comment

rallytime commented Jul 31, 2018

anlutro commented Jul 31, 2018

anlutro commented Jul 31, 2018

terminalmage left a comment •

edited

Loading

anlutro commented Aug 20, 2018

terminalmage commented Aug 20, 2018

isbm left a comment

cachedout commented Aug 28, 2018

anlutro commented Aug 28, 2018 via email

rallytime commented Aug 28, 2018

terminalmage left a comment

anlutro commented Aug 29, 2018

terminalmage commented Aug 29, 2018

rallytime commented Aug 30, 2018

cachedout commented Oct 16, 2018

anlutro commented Oct 16, 2018

terminalmage left a comment

cachedout commented Dec 10, 2018

terminalmage commented Dec 10, 2018

cachedout commented Dec 10, 2018

terminalmage commented Dec 11, 2018

More intelligent diffing in changes of file.serialize state #48609

More intelligent diffing in changes of file.serialize state #48609

Conversation

anlutro commented Jul 16, 2018

DmitryKuzmenko commented Jul 27, 2018

anlutro commented Jul 28, 2018

DmitryKuzmenko commented Jul 30, 2018

anlutro commented Jul 30, 2018

rallytime left a comment

Choose a reason for hiding this comment

rallytime Jul 30, 2018

Choose a reason for hiding this comment

terminalmage left a comment

Choose a reason for hiding this comment

rallytime commented Jul 31, 2018

anlutro commented Jul 31, 2018

anlutro commented Jul 31, 2018

terminalmage left a comment • edited Loading

Choose a reason for hiding this comment

anlutro commented Aug 20, 2018

terminalmage commented Aug 20, 2018

isbm left a comment

Choose a reason for hiding this comment

cachedout commented Aug 28, 2018

anlutro commented Aug 28, 2018 via email

rallytime commented Aug 28, 2018

terminalmage left a comment

Choose a reason for hiding this comment

anlutro commented Aug 29, 2018

terminalmage commented Aug 29, 2018

rallytime commented Aug 30, 2018

cachedout commented Oct 16, 2018

anlutro commented Oct 16, 2018

terminalmage left a comment

Choose a reason for hiding this comment

cachedout commented Dec 10, 2018

terminalmage commented Dec 10, 2018

cachedout commented Dec 10, 2018

terminalmage commented Dec 11, 2018

terminalmage left a comment •

edited

Loading