Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

update to ownCloud 9.1.1 (with intermediate upgrades) #894

Merged
merged 15 commits into from
Oct 18, 2016
Merged

update to ownCloud 9.1.1 (with intermediate upgrades) #894

Show file tree
Hide file tree
Changes from 14 commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
e635b5e
Install owncoud 9.1 and provide an upgrade path from 8.2. This also d…
yodax Jul 25, 2016
2ac25b5
Remove php apc setting
yodax Jul 27, 2016
fa5c4fb
Add dav migrations for each user
yodax Jul 27, 2016
f1ec1eb
Add some comments to the code
yodax Jul 27, 2016
abe6bc7
When upgrading owncloud from 8.2.3 to 9.1.0 the backup of 8.2.3 was o…
yodax Jul 29, 2016
3f65c71
Add upgrade path from 8.1.1. Only do an upgrade check if owncloud was…
yodax Jul 30, 2016
665d5fc
Stop php5-fpm before owncloud upgrade to prevent database locks
yodax Jul 31, 2016
294f13c
Merge branch 'master' into owncloud9
yodax Jul 31, 2016
c5f7af8
Fix fail2ban tests for owncloud 9
yodax Jul 31, 2016
d6bd7f7
When upgrading owncloud copy the database to the user-data/owncloud-b…
yodax Aug 1, 2016
9a3742f
Merge with master, test owncloud upgrade from 8.2.7
yodax Aug 14, 2016
f9451ce
Merge with master, for conflicts on tests/fail2ban.py
yodax Sep 18, 2016
7c0b4fa
Remove not need unzip directives during owncloud extraction. Director…
yodax Oct 8, 2016
1b63708
Improve backup of owncloud installation and provide a post installati…
yodax Oct 16, 2016
8bf42a6
Separate the ownCloud upgrades visually in the console output.
yodax Oct 17, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
113 changes: 89 additions & 24 deletions setup/owncloud.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,10 +16,6 @@ apt_install \

apt-get purge -qq -y owncloud*

# Install ownCloud from source of this version:
owncloud_ver=8.2.7
owncloud_hash=723ba3f46dad219109cdf28dcc016fcd8a6bc434

# Migrate <= v0.10 setups that stored the ownCloud config.php in /usr/local rather than
# in STORAGE_ROOT. Move the file to STORAGE_ROOT.
if [ ! -f $STORAGE_ROOT/owncloud/config.php ] \
Expand All @@ -32,28 +28,30 @@ if [ ! -f $STORAGE_ROOT/owncloud/config.php ] \
ln -sf $STORAGE_ROOT/owncloud/config.php /usr/local/lib/owncloud/config/config.php
fi

# Check if ownCloud dir exist, and check if version matches owncloud_ver (if either doesn't - install/upgrade)
if [ ! -d /usr/local/lib/owncloud/ ] \
|| ! grep -q $owncloud_ver /usr/local/lib/owncloud/version.php; then
InstallOwncloud() {
version=$1
hash=$2
Copy link
Member

@JoshData JoshData Oct 16, 2016 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would be good to add something like echo Upgrading to ownCloud $version... here, so that when there are multiple upgrades happening the console output is clear about what's happening when.


# Download and verify
wget_verify https://download.owncloud.org/community/owncloud-$owncloud_ver.zip $owncloud_hash /tmp/owncloud.zip
# Remove the current owncloud
rm -rf /usr/local/lib/owncloud
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

According to the comment at the unzip command this should not be necessary. Maybe remove or clarify the comment?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment on the unzip has always been incorrect, previously the directory was moved. It can indeed be changed for clarity.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the -u option from the unzip command can go as well.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, it's probably better to delete first, especially as otherwise files that are in the old ownCloud release but have been removed in a newer version would remain. In that case the -u -f and -o can go altogether, since there would be no existing files at that point.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I pushed a change for this. Retested and it works.


# Clear out the existing ownCloud.
if [ -d /usr/local/lib/owncloud/ ]; then
echo "upgrading ownCloud to $owncloud_ver (backing up existing ownCloud directory to /tmp/owncloud-backup-$$)..."
mv /usr/local/lib/owncloud /tmp/owncloud-backup-$$
fi
# Download and verify
wget_verify https://download.owncloud.org/community/owncloud-$version.zip $hash /tmp/owncloud.zip

# Extract ownCloud
unzip -u -o -q /tmp/owncloud.zip -d /usr/local/lib #either extracts new or replaces current files
unzip -q /tmp/owncloud.zip -d /usr/local/lib
rm -f /tmp/owncloud.zip

# The two apps we actually want are not in ownCloud core. Clone them from
# The two apps we actually want are not in ownCloud core. Download the releases from
# their github repositories.
mkdir -p /usr/local/lib/owncloud/apps
git_clone https://github.com/owncloudarchive/contacts 9ba2e667ae8c7ea36d8c4a4c3413c374beb24b1b '' /usr/local/lib/owncloud/apps/contacts
git_clone https://github.com/owncloudarchive/calendar 2086e738a3b7b868ec59cd61f0f88b49c3f21dd1 '' /usr/local/lib/owncloud/apps/calendar
wget_verify https://github.com/owncloud/contacts/releases/download/v1.4.0.0/contacts.tar.gz c1c22d29699456a45db447281682e8bc3f10e3e7 /tmp/contacts.tgz
tar xf /tmp/contacts.tgz -C /usr/local/lib/owncloud/apps/
rm /tmp/contacts.tgz

wget_verify https://github.com/nextcloud/calendar/releases/download/v1.4.0/calendar.tar.gz c84f3170efca2a99ea6254de34b0af3cb0b3a821 /tmp/calendar.tgz
tar xf /tmp/calendar.tgz -C /usr/local/lib/owncloud/apps/
rm /tmp/calendar.tgz

# Fix weird permissions.
chmod 750 /usr/local/lib/owncloud/{apps,config}
Expand All @@ -69,7 +67,7 @@ if [ ! -d /usr/local/lib/owncloud/ ] \

# If this isn't a new installation, immediately run the upgrade script.
# Then check for success (0=ok and 3=no upgrade needed, both are success).
if [ -f $STORAGE_ROOT/owncloud/owncloud.db ]; then
if [ -e $STORAGE_ROOT/owncloud/owncloud.db ]; then
# ownCloud 8.1.1 broke upgrades. It may fail on the first attempt, but
# that can be OK.
sudo -u www-data php /usr/local/lib/owncloud/occ upgrade
Expand All @@ -81,6 +79,76 @@ if [ ! -d /usr/local/lib/owncloud/ ] \
echo "...which seemed to work."
fi
fi
}

owncloud_ver=9.1.1

# Check if ownCloud dir exist, and check if version matches owncloud_ver (if either doesn't - install/upgrade)
if [ ! -d /usr/local/lib/owncloud/ ] \
|| ! grep -q $owncloud_ver /usr/local/lib/owncloud/version.php; then

# Stop php-fpm
hide_output service php5-fpm stop
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this really needed? php should not need a restart even if all the files are changed.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See my comment above:

Also I found a problem with the owncloud upgrades when somebody was trying to use owncloud at the same time. Before an upgrade I stop the service, this solved the issue.


# Backup the existing ownCloud.
# Create a backup directory to store the current installation and database to
BACKUP_DIRECTORY=$STORAGE_ROOT/owncloud-backup/`date +"%Y-%m-%d-%T"`
mkdir -p "$BACKUP_DIRECTORY"
if [ -d /usr/local/lib/owncloud/ ]; then
echo "upgrading ownCloud to $owncloud_ver (backing up existing ownCloud installation, configuration and database to directory to $BACKUP_DIRECTORY..."
cp -r /usr/local/lib/owncloud "$BACKUP_DIRECTORY/owncloud-install"
fi
if [ -e /home/user-data/owncloud/owncloud.db ]; then
cp /home/user-data/owncloud/owncloud.db $BACKUP_DIRECTORY
fi
if [ -e /home/user-data/owncloud/config.php ]; then
cp /home/user-data/owncloud/config.php $BACKUP_DIRECTORY
fi

# We only need to check if we do upgrades when owncloud was previously installed
if [ -e /usr/local/lib/owncloud/version.php ]; then
if grep -q "8.1.[0-9]" /usr/local/lib/owncloud/version.php; then
echo "We are running 8.1.x, upgrading to 8.2.3 first"
InstallOwncloud 8.2.3 bfdf6166fbf6fc5438dc358600e7239d1c970613
fi

# If we are upgrading from 8.2.x we should go to 9.0 first. Owncloud doesn't support skipping minor versions
if grep -q "8.2.[0-9]" /usr/local/lib/owncloud/version.php; then
echo "We are running version 8.2.x, upgrading to 9.0.2 first"

# We need to disable memcached. The upgrade and install fails
# with memcached
CONFIG_TEMP=$(/bin/mktemp)
php <<EOF > $CONFIG_TEMP && mv $CONFIG_TEMP $STORAGE_ROOT/owncloud/config.php;
<?php
include("$STORAGE_ROOT/owncloud/config.php");

\$CONFIG['memcache.local'] = '\OC\Memcache\APC';

echo "<?php\n\\\$CONFIG = ";
var_export(\$CONFIG);
echo ";";
?>
EOF
chown www-data.www-data $STORAGE_ROOT/owncloud/config.php

# We can now install owncloud 9.0.2
InstallOwncloud 9.0.2 72a3d15d09f58c06fa8bee48b9e60c9cd356f9c5

# The owncloud 9 migration doesn't migrate calendars and contacts
# The option to migrate these are removed in 9.1
# So the migrations should be done when we have 9.0 installed
sudo -u www-data php /usr/local/lib/owncloud/occ dav:migrate-addressbooks
# The following migration has to be done for each owncloud user
for directory in $STORAGE_ROOT/owncloud/*@*/ ; do
username=$(basename "${directory}")
sudo -u www-data php /usr/local/lib/owncloud/occ dav:migrate-calendar $username
done
sudo -u www-data php /usr/local/lib/owncloud/occ dav:sync-birthday-calendar
fi
fi

InstallOwncloud $owncloud_ver 72ed9812432f01b3a459c4afc33f5c76b71eec09
fi

# ### Configuring ownCloud
Expand Down Expand Up @@ -110,10 +178,7 @@ if [ ! -f $STORAGE_ROOT/owncloud/owncloud.db ]; then
'arguments'=>array('{127.0.0.1:993/imap/ssl/novalidate-cert}')
)
),
'memcache.local' => '\\OC\\Memcache\\Memcached',
"memcached_servers" => array (
array('127.0.0.1', 11211),
),
'memcache.local' => '\OC\Memcache\APC',
'mail_smtpmode' => 'sendmail',
'mail_smtpsecure' => '',
'mail_smtpauthtype' => 'LOGIN',
Expand Down Expand Up @@ -173,7 +238,7 @@ include("$STORAGE_ROOT/owncloud/config.php");

\$CONFIG['trusted_domains'] = array('$PRIMARY_HOSTNAME');

\$CONFIG['memcache.local'] = '\\OC\\Memcache\\Memcached';
\$CONFIG['memcache.local'] = '\OC\Memcache\APC';
\$CONFIG['overwrite.cli.url'] = '/cloud';
\$CONFIG['mail_from_address'] = 'administrator'; # just the local part, matches our master administrator address

Expand Down
8 changes: 4 additions & 4 deletions tests/fail2ban.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,11 +10,11 @@

# parse command line

if len(sys.argv) != 3:
print("Usage: tests/fail2ban.py \"ssh user@hostname\" hostname")
if len(sys.argv) != 4:
print("Usage: tests/fail2ban.py \"ssh user@hostname\" hostname owncloud_user")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are these changes needed for OC 9 or are they additional improvements? If it is the second I would suggest splitting them up into another PR.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is to test that fail2ban still works, so yes it's related. See discussion above.

sys.exit(1)

ssh_command, hostname = sys.argv[1:3]
ssh_command, hostname, owncloud_user = sys.argv[1:4]

# define some test types

Expand Down Expand Up @@ -215,7 +215,7 @@ def run_test(testfunc, args, count, within_seconds, parallel):
run_test(http_test, ["/admin/munin/", 401], 20, 30, 1)

# ownCloud
run_test(http_test, ["/cloud/remote.php/webdav", 401, None, None, ["aa", "aa"]], 20, 120, 1)
run_test(http_test, ["/cloud/remote.php/webdav", 401, None, None, [owncloud_user, "aa"]], 20, 120, 1)

# restart fail2ban so that this client machine is no longer blocked
restart_fail2ban_service(final=True)
49 changes: 49 additions & 0 deletions tools/owncloud-restore.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
#!/bin/bash
#
# This script will restore the backup made during an installation
source /etc/mailinabox.conf # load global vars

if [ -z "$1" ]; then
echo "Usage: owncloud-restore.sh <backup directory>"
echo
echo "WARNING: This will restore the database to the point of the installation!"
echo " This means that you will lose all changes made by users after that point"
echo
echo
echo "Backups are stored here: $STORAGE_ROOT/owncloud-backup/"
echo
echo "Available backups:"
echo
find $STORAGE_ROOT/owncloud-backup/* -maxdepth 0 -type d
echo
echo "Supply the directory that was created during the last installation as the only commandline argument"
exit
fi

if [ ! -f $1/config.php ]; then
echo "This isn't a valid backup location"
exit
fi

echo "Restoring backup from $1"
service php5-fpm stop

# remove the current owncloud installation
rm -rf /usr/local/lib/owncloud/
# restore the current owncloud application
cp -r "$1/owncloud-install" /usr/local/lib/owncloud

# restore access rights
chmod 750 /usr/local/lib/owncloud/{apps,config}

cp "$1/owncloud.db" $STORAGE_ROOT/owncloud/
cp "$1/config.php" $STORAGE_ROOT/owncloud/

ln -sf $STORAGE_ROOT/owncloud/config.php /usr/local/lib/owncloud/config/config.php
chown -f -R www-data.www-data $STORAGE_ROOT/owncloud /usr/local/lib/owncloud
chown www-data.www-data $STORAGE_ROOT/owncloud/config.php

sudo -u www-data php /usr/local/lib/owncloud/occ maintenance:mode --off

service php5-fpm start
echo "Done"