Failure to cherrypick changelog back to master

[daviddavis]

We encountered a problem when releasing 3.14.1 when trying to cherrypick the changelog back to master:

https://github.com/pulp/pulpcore/runs/3011763886?check_suite_focus=true#step:23:37

I believe the problem is that due to a z-stream release, the changelog commit is referencing (and trying to remove) changelog files that don't exist on master. For example, there's CHANGES/9015.bugfix which is the 3.14.1 backport of CHANGES/9003.bugfix.

[daviddavis]

I started a related discussion here: https://github.com/pulp/community/discussions/50.