[4.0] Improve the generator tool for lists of files and folders to be removed or renamed on update

[richard67]

Pull Request for Issue # .

Replaces #21 .

Summary of Changes

This pull request (PR) is just to document the changed script "build/deleted_file_check.php" which I am currently using to generate the lists of deleted files and folders and renamed files for updating script.php.

The changes in the script are:

• Support zip files as parameters - Zip files will not be unpacked for the check. Instead of this, the files and folders lists and the version number from the manifest XML are directly read from the zip file.
• Let the "to" parameter for the ending version be optional like it was intended to be but up to now not had been implemented.
  If the "to" parameter is not given, a full package zip from a previous run of the build script will be used if such a file exists.
• Not create the result files but append the results of the current comparison to the particular files if they aren't already included in that file from previous comparisons.
  This allows to execute a sequence of comparisons, see description below on how I use the changed script with a shell script to completely rebuild the lists.
• When a file or a folder once had been removed so it is included in results of previous comparisons and now with the current "to" version is added back, it is removed from the previous results.
  For an example see [4.0] Remove _com_joomlaupdate.scss from deleted files in script.php joomla/joomla-cms#36161 .
• Prepend each comparison with a comment which contains the from and to version.
  The version numbers are read from the manifest XML files "administrator/manifests/files/joomla.xml" of the "from" and "to" directories or zip packages.
• Update docs and comments to reflect all previous changes as well as up to now undocumented changes in the 4.0-dev branch.

Testing Instructions

Currently I use following Bash script to generate the lists for the 4.0-dev branch:

#!/usr/bin/env bash

BASEDIR=$(dirname "$(realpath $0)")
cd $BASEDIR

mkdir -p ./tmp/deleted_files_check/

# Build current version if there is no result present from a previous build
if [ ! -f ./tmp/packages/*Full_Package.zip ]
then
	php ./build.php --remote=HEAD --exclude-gzip --exclude-bzip2
fi

CURRENTZIP=`ls -1 ./tmp/packages/*Full_Package.zip`

# Clone and build previous major version if there is no result present from a previous build
if [ ! -f ./tmp/deleted_files_check/previous-package/*Full_Package.zip ]
then
	mkdir -p ./tmp/deleted_files_check/previous-package
	mkdir -p ./tmp/deleted_files_check/previous-build
	cd ./tmp/deleted_files_check/previous-build
	git clone -b 3.10-dev https://github.com/joomla/joomla-cms.git .
	php ./build/build.php --remote=HEAD --exclude-gzip --exclude-bzip2
	cd $BASEDIR
	if [ -f ./tmp/deleted_files_check/previous-build/build/tmp/packages/*Full_Package.zip ]
	then
		cp ./tmp/deleted_files_check/previous-build/build/tmp/packages/*Full_Package.zip ./tmp/deleted_files_check/previous-package/
	fi
fi

rm -rf ./tmp/deleted_files_check/previous-build

PREVIOUSZIP=`ls -1 ./tmp/deleted_files_check/previous-package/*Full_Package.zip`

# Start with comparing the previous major release build with the current version's build.
rm -f ./deleted_files.txt
rm -f ./deleted_folders.txt
rm -f ./renamed_files.txt
php ./deleted_file_check.php --from=$PREVIOUSZIP --to=$CURRENTZIP

# Compare past beta or RC or stable releases of the current version from old to new
php ./deleted_file_check.php \
  --from=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.0-beta1-Beta-Full_Package.zip \
  --to=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.0-beta2-Beta-Full_Package.zip
php ./deleted_file_check.php \
  --from=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.0-beta2-Beta-Full_Package.zip \
  --to=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.0-beta3-Beta-Full_Package.zip
php ./deleted_file_check.php \
  --from=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.0-beta3-Beta-Full_Package.zip \
  --to=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.0-beta4-Beta-Full_Package.zip
php ./deleted_file_check.php \
  --from=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.0-beta4-Beta-Full_Package.zip \
  --to=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.0-beta5-Beta-Full_Package.zip
php ./deleted_file_check.php \
  --from=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.0-beta5-Beta-Full_Package.zip \
  --to=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.0-beta6-Beta-Full_Package.zip
php ./deleted_file_check.php \
  --from=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.0-beta6-Beta-Full_Package.zip \
  --to=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.0-beta7-Beta-Full_Package.zip
php ./deleted_file_check.php \
  --from=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.0-beta7-Beta-Full_Package.zip \
  --to=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.0-rc1-Release_Candidate-Full_Package.zip
php ./deleted_file_check.php \
  --from=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.0-rc1-Release_Candidate-Full_Package.zip \
  --to=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.0-rc2-Release_Candidate-Full_Package.zip
php ./deleted_file_check.php \
  --from=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.0-rc2-Release_Candidate-Full_Package.zip \
  --to=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.0-rc3-Release_Candidate-Full_Package.zip
php ./deleted_file_check.php \
  --from=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.0-rc3-Release_Candidate-Full_Package.zip \
  --to=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.0-rc4-Release_Candidate-Full_Package.zip
php ./deleted_file_check.php \
  --from=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.0-rc4-Release_Candidate-Full_Package.zip \
  --to=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.0-rc5-Release_Candidate-Full_Package.zip
php ./deleted_file_check.php \
  --from=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.0-rc5-Release_Candidate-Full_Package.zip \
  --to=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.0-rc6-Release_Candidate-Full_Package.zip
php ./deleted_file_check.php \
  --from=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.0-rc6-Release_Candidate-Full_Package.zip \
  --to=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.0-Stable-Full_Package.zip
php ./deleted_file_check.php \
  --from=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.0-Stable-Full_Package.zip \
  --to=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.1-Stable-Full_Package.zip
php ./deleted_file_check.php \
  --from=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.1-Stable-Full_Package.zip \
  --to=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.2-Stable-Full_Package.zip
php ./deleted_file_check.php \
  --from=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.2-Stable-Full_Package.zip \
  --to=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.3-Stable-Full_Package.zip
php ./deleted_file_check.php \
  --from=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.3-Stable-Full_Package.zip \
  --to=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.4-Stable-Full_Package.zip
php ./deleted_file_check.php \
  --from=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.4-Stable-Full_Package.zip \
  --to=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.5-Stable-Full_Package.zip

# Finish with comarison between the latest release and the current build
php ./deleted_file_check.php \
  --from=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.5-Stable-Full_Package.zip \
  --to=$CURRENTZIP

When keeping the intermediate results from comparing older 4.0 versions since Beta 1, it only needs to do the first comparison from current 3.10-dev to current 4.0-dev and then the last 2 comparisons. The 2nd last because something might have been forgotten in script.php when the last release was made, and the last one for the last release and the current branch.

At the end I can take the complete content of each file, intend it by 3 tabs and copy it into script PHP and can see the changes in Git.

Then it needs only 3 manual steps:

• Add back the intermediate 4.0.0 update SQL scripts which I had removed between 2 beta versions when combining the update SQL script, so the comparisons don't know about them.
• Move deletion of folder "/libraries/vendor/paragonie/random_compat" from section "// From 3.10 to 4.0" to section "// 4.0 from Beta 7 to RC 1". The reason is that we have different child folders of that being deleted when updating from 3.10 or when updating from 4.0 Beta 7, so we get the folder 2 times, and the duplicate check added in this PR here will add it do the 1st and not to the 2nd section, but in this case it should be in the 2nd.
• Remove section comments for empty sections

This can be automated later with further improvements, but the question now is if what I am doing here is the right way to go.

For 4.1-dev I run the following shell script after having run the previously mentioned one for 4.0-dev:

#!/usr/bin/env bash

# Copy results from 4.0-dev
cp -f /home/richard/lamp/public_html/joomla-cms-4.0-dev/build/deleted_files.txt ./
cp -f /home/richard/lamp/public_html/joomla-cms-4.0-dev/build/deleted_folders.txt ./
cp -f /home/richard/lamp/public_html/joomla-cms-4.0-dev/build/renamed_files.txt ./

# From 4.0-dev to 4.1-dev - ignore results from previous comparisons
php ./deleted_file_check.php \
    --from=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.5-rc5-dev-Development-Full_Package \
    --to=/home/richard/lamp/public_html/4.1-compare-releases/Joomla_4.1.0-dev-Development-Full_Package

Actual result BEFORE applying this Pull Request

I have to download or build full package zip files and then unpack them for all comparisons.

I have to identify and remove duplicates myself from the right section of the deleted files and folder lists in script.php.

I have to remove files or folders from the lists which once had been removed and later been added back between releases.

The help and the documentation in the generator tool are wrong.

Expected result AFTER applying this Pull Request

I don't have to unpack full package zip files.

If the "to" parameter is not given, the full package from a previous run of the build script is used, if present.

Files and folders which have been identified to be removed when updating from the previous major (or minor) release on another branch, e.g. in case of J4 from 3.10, are not to be removed again when updating from a previous version of the current release to be checked.

When some file or folder is removed between 2 versions and later added back with another, later version, the tool is able to handle that yet and remove them from the lists.

The documentation and help in the generator tool describe what is implemented.

Documentation Changes Required

There is no documentation yet on how to use deleted_file_check.php.