testutil/pkgdata: Create deb programatically

Currently we only test with base-files deb which is embedded as base64
encoded binary in testutil/pkgdata.go. If we want to change the contents
of the data files, we have to un-ar it, un-tar it, change the data
files, tar it and ar it. Furthermore, these operations modify attributes
based on the workstation on which this process is run. So we end up with
arbitrary user/group ids/names names and dates.

Fix it by creating debs programatically. The new interface provides way
to create debs with arbitrary content of data tarball.

internal/testutil/pkgdata/base-files/etc/debian_version

@@ -0,0 +1 @@
+bullseye/sid

internal/testutil/pkgdata/base-files/etc/dpkg/origins/debian

@@ -0,0 +1,3 @@
+Vendor: Debian
+Vendor-URL: http://www.debian.org/
+Bugs: debbugs://bugs.debian.org

internal/testutil/pkgdata/base-files/etc/dpkg/origins/ubuntu

@@ -0,0 +1,4 @@
+Vendor: Ubuntu
+Vendor-URL: http://www.ubuntu.com/
+Bugs: https://bugs.launchpad.net/ubuntu/+filebug
+Parent: Debian

internal/testutil/pkgdata/base-files/etc/host.conf

@@ -0,0 +1,3 @@
+# The "order" line is only used by old versions of the C library.
+order hosts,bind
+multi on

internal/testutil/pkgdata/base-files/etc/issue

@@ -0,0 +1,2 @@
+Ubuntu 20.04.4 LTS \n \l
+

internal/testutil/pkgdata/base-files/etc/issue.net

@@ -0,0 +1 @@
+Ubuntu 20.04.4 LTS

internal/testutil/pkgdata/base-files/etc/legal

@@ -0,0 +1,8 @@
+
+The programs included with the Ubuntu system are free software;
+the exact distribution terms for each program are described in the
+individual files in /usr/share/doc/*/copyright.
+
+Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
+applicable law.
+

internal/testutil/pkgdata/base-files/etc/lsb-release

@@ -0,0 +1,4 @@
+DISTRIB_ID=Ubuntu
+DISTRIB_RELEASE=20.04
+DISTRIB_CODENAME=focal
+DISTRIB_DESCRIPTION="Ubuntu 20.04.4 LTS"

internal/testutil/pkgdata/base-files/etc/profile.d/01-locale-fix.sh

@@ -0,0 +1,2 @@
+# Make sure the locale variables are set to valid values.
+eval $(/usr/bin/locale-check C.UTF-8)

internal/testutil/pkgdata/base-files/etc/update-motd.d/00-header

@@ -0,0 +1,29 @@
+#!/bin/sh
+#
+#    00-header - create the header of the MOTD
+#    Copyright (C) 2009-2010 Canonical Ltd.
+#
+#    Authors: Dustin Kirkland <[email protected]>
+#
+#    This program is free software; you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License as published by
+#    the Free Software Foundation; either version 2 of the License, or
+#    (at your option) any later version.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License along
+#    with this program; if not, write to the Free Software Foundation, Inc.,
+#    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+[ -r /etc/lsb-release ] && . /etc/lsb-release
+
+if [ -z "$DISTRIB_DESCRIPTION" ] && [ -x /usr/bin/lsb_release ]; then
+	# Fall back to using the very slow lsb_release utility
+	DISTRIB_DESCRIPTION=$(lsb_release -s -d)
+fi
+
+printf "Welcome to %s (%s %s %s)\n" "$DISTRIB_DESCRIPTION" "$(uname -o)" "$(uname -r)" "$(uname -m)"

internal/testutil/pkgdata/base-files/etc/update-motd.d/10-help-text

@@ -0,0 +1,26 @@
+#!/bin/sh
+#
+#    10-help-text - print the help text associated with the distro
+#    Copyright (C) 2009-2010 Canonical Ltd.
+#
+#    Authors: Dustin Kirkland <[email protected]>,
+#             Brian Murray <[email protected]>
+#
+#    This program is free software; you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License as published by
+#    the Free Software Foundation; either version 2 of the License, or
+#    (at your option) any later version.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License along
+#    with this program; if not, write to the Free Software Foundation, Inc.,
+#    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+printf "\n"
+printf " * Documentation:  https://help.ubuntu.com\n"
+printf " * Management:     https://landscape.canonical.com\n"
+printf " * Support:        https://ubuntu.com/advantage\n"

internal/testutil/pkgdata/base-files/etc/update-motd.d/50-motd-news

@@ -0,0 +1,146 @@
+#!/bin/sh
+#
+#    50-motd-news - print the live news from the Ubuntu wire
+#    Copyright (C) 2016-2020 Canonical Ltd.
+#    Copyright (C) 2016-2017 Dustin Kirkland
+#
+#    Authors: Dustin Kirkland <[email protected]>
+#             Steve Langasek <[email protected]>
+#
+#    This program is free software; you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License as published by
+#    the Free Software Foundation; either version 2 of the License, or
+#    (at your option) any later version.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License along
+#    with this program; if not, write to the Free Software Foundation, Inc.,
+#    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+##############################################################################
+# This program could be rewritten in C or Golang for faster performance.
+# Or it could be rewritten in Python or another higher level language
+# for more modularity.
+# However, I've insisted on shell here for transparency!
+#                                                                     - Dustin
+##############################################################################
+
+# Source the local configuration
+[ -r /etc/default/motd-news ] && . /etc/default/motd-news
+
+# Exit immediately, unless we're enabled
+# This makes this script very easy to disable in /etc/default/motd-news configuration
+[ "$ENABLED" = "1" ] || exit 0
+
+# Ensure sane defaults
+[ -n "$URLS" ] || URLS="https://motd.ubuntu.com"
+[ -n "$WAIT" ] || WAIT=5
+[ -n "$CACHE" ] || CACHE="/var/cache/motd-news"
+[ "$1" = "--force" ] && FORCED=1
+
+# Ensure we print safely, maximum of the first 10 lines,
+# maximum of the first 80 chars per line, no control chars
+safe_print() {
+	cat "$1" | head -n 10 | tr -d '\000-\011\013\014\016-\037' | cut -c -80
+}
+
+
+# If we're not forcing an update, and we have a cached motd-news file,
+# then just print it and exit as quickly as possible, for login performance.
+# Note that systemd should keep this cache file up to date, asynchronously
+if [ "$FORCED" != "1" ]; then
+  	if [ -r $CACHE ]; then
+		echo
+		safe_print $CACHE
+	elif [ "$(id -u)" -eq 0 ]; then
+		: > $CACHE
+	fi
+	exit 0
+fi
+
+# If we've made it here, we've been given the --force argument,
+# probably from the systemd motd-news.service.  Let's update...
+
+# Abort early if wget is missing
+[ -x /usr/bin/wget ] || exit 0
+
+# Generate our temp files, clean up when done
+NEWS=$(mktemp) || exit 1
+ERR=$(mktemp) || exit 1
+CLOUD=$(mktemp) || exit 1
+trap "rm -f $NEWS $ERR $CLOUD" HUP INT QUIT ILL TRAP KILL BUS TERM
+
+# Construct a user agent, similar to Firefox/Chrome/Safari/IE to
+# ensure a proper, tailored, accurate message of the day
+
+# wget browser version, for debug purposes
+wget_ver="$(dpkg -l wget | awk '$1 == "ii" { print($3); exit(0); }')"
+
+# Distribution version, for messages releated to this Ubuntu release
+. /etc/lsb-release
+lsb=$(echo "$DISTRIB_DESCRIPTION" | sed -e "s/ /\//g")
+codename="$DISTRIB_CODENAME"
+
+# Kernel version and CPU type, for messages related to a particular revision or hardware
+platform="$(uname -o)/$(uname -r)/$(uname -m)"
+arch="$(uname -m)"
+cpu="$(grep -m1 "^model name" /proc/cpuinfo | sed -e "s/.*: //" -e "s:\s\+:/:g")"
+cloud_id="unknown"
+if [ -x /usr/bin/cloud-id ]; then
+    /usr/bin/cloud-id > "$CLOUD" 2>/dev/null
+    if [ "$?" -eq "0" ]; then
+        # sanitize it a bit, just in case
+        cloud_id=$(cut -c -40 "${CLOUD}" | tr -c -d '[:alnum:]')
+        if [ -z "${cloud_id}" ]; then
+            cloud_id="unknown"
+        fi
+    fi
+fi
+
+# Piece together the user agent
+USER_AGENT="wget/$wget_ver $lsb $platform $cpu cloud_id/$cloud_id"
+
+# Loop over any configured URLs
+for u in $URLS; do
+	# Ensure https:// protocol, for security reasons
+	case $u in
+		https://*)
+			true
+		;;
+		https://motd.ubuntu.com)
+			u="$u/$codename/$arch"
+		;;
+		*)
+			continue
+		;;
+	esac
+	# If we're forced, set the wait to much higher (1 minute)
+	[ "$FORCED" = "1" ] && WAIT=60
+	# Fetch and print the news motd
+	result=0
+	not_found_is_ok=0
+	wget --timeout "$WAIT" -U "$USER_AGENT" -O- --content-on-error "$u" >"$NEWS" 2>"$ERR" || result=$?
+	# from wget's manpage: 8   Server issued an error response.
+	if [ $result -eq 8 ]; then
+		if grep -q "ERROR 404" "$ERR"; then
+			# The server's 404 document is the generic, non cloud-specific, motd-news
+			# content present in the index.txt file
+			not_found_is_ok=1
+		fi
+	fi
+	if [ $result -eq 0 ] || [ $not_found_is_ok -eq 1 ]; then
+		echo
+		# At most, 10 lines of text, remove control characters, print at most 80 characters per line
+		safe_print "$NEWS"
+		# Try to update the cache
+		safe_print "$NEWS" 2>/dev/null >$CACHE || true
+	else
+		: > "$CACHE"
+ 	fi
+done
+rm -f "$NEWS" "$ERR" "$CLOUD"
+exit 0

internal/testutil/pkgdata/base-files/lib/systemd/system/motd-news.service

@@ -0,0 +1,8 @@
+[Unit]
+Description=Message of the Day
+After=network-online.target
+Documentation=man:update-motd(8)
+
+[Service]
+Type=oneshot
+ExecStart=/etc/update-motd.d/50-motd-news --force

internal/testutil/pkgdata/base-files/lib/systemd/system/motd-news.timer

@@ -0,0 +1,11 @@
+[Unit]
+Description=Message of the Day
+
+[Timer]
+OnCalendar=00,12:00:00
+RandomizedDelaySec=12h
+Persistent=true
+OnStartupSec=1min
+
+[Install]
+WantedBy=timers.target

internal/testutil/pkgdata/base-files/usr/bin/hello

@@ -0,0 +1,2 @@
+#!/bin/sh
+echo "Hello world"

internal/testutil/pkgdata/base-files/usr/lib/os-release

@@ -0,0 +1,12 @@
+NAME="Ubuntu"
+VERSION="20.04.4 LTS (Focal Fossa)"
+ID=ubuntu
+ID_LIKE=debian
+PRETTY_NAME="Ubuntu 20.04.4 LTS"
+VERSION_ID="20.04"
+HOME_URL="https://www.ubuntu.com/"
+SUPPORT_URL="https://help.ubuntu.com/"
+BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
+PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
+VERSION_CODENAME=focal
+UBUNTU_CODENAME=focal

internal/testutil/pkgdata/base-files/usr/share/doc/base-files/copyright

@@ -0,0 +1,28 @@
+This is the Debian GNU/Linux prepackaged version of the Debian Base System
+Miscellaneous files. These files were written by Ian Murdock
+<[email protected]> and Bruce Perens <[email protected]>.
+
+This package was first put together by Bruce Perens <[email protected]>,
+from his own sources.
+
+The GNU Public Licenses in /usr/share/common-licenses were taken from
+ftp.gnu.org and are copyrighted by the Free Software Foundation, Inc.
+
+The Artistic License in /usr/share/common-licenses is the one coming
+from Perl and its SPDX name is "Artistic License 1.0 (Perl)".
+
+
+Copyright (C) 1995-2011 Software in the Public Interest.
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+On Debian GNU/Linux systems, the complete text of the GNU General
+Public License can be found in `/usr/share/common-licenses/GPL'.