Add state flag file in K8s upgrades (#2702)

[cicharka]

• it is based on solution under upgrade/tasks/kafka.yml
• added usage of update flag for masters and nodes - having single
  flag file for both was introducing malfunction (incomplete master
  upgrade flag was removed by tasks in nodes block)
• 'Switch apiserver address to local' task is now executed before the
  upgrade procedure in order to correctly fetch versions of particular
  masters - without it procedure for some masters could be skipped,
  because fetched version didn't match host it tried to upgrade
• separate get and set cluster version tasks
• introduce initial_cluster_version used to evaluate need of
  an upgrade

Tested on cluster (Ubuntu machines) containing:

1. three k8s masters and three k8s nodes (HA) (upgrade from v1.0.1 to v1.3.0)

[atsikham]

Have you checked creation of files for each incremental upgrade?

[atsikham]

Just FYI, this PR should fix such issue that I met recently.

TASK [upgrade : k8s/install | Install kubeadm 1.20.12 package for Debian family] *************************************************************************
fatal: [ec2-18-212-2-237.compute-1.amazonaws.com]: FAILED! => {"cache_update_time": 1639014589, "cache_updated": true, "changed": false, "msg": "'/usr/bin/apt-get -y -o \"Dpkg::Options::=--force-confdef\" -o \"Dpkg::Options::=--force-confold\"      install 'kubeadm=1.20.12-00' 'kubelet=1.20.12-00'' failed: E: Packages were downgraded and -y was used without --allow-downgrades.\n", "rc": 100, "stderr": "E: Packages were downgraded and -y was used without --allow-downgrades.\n", "stderr_lines": ["E: Packages were downgraded and -y was used without --allow-downgrades."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages will be DOWNGRADED:\n  kubeadm kubelet\n0 upgraded, 0 newly installed, 2 downgraded, 0 to remove and 56 not upgraded.\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages will be DOWNGRADED:", "  kubeadm kubelet", "0 upgraded, 0 newly installed, 2 downgraded, 0 to remove and 56 not upgraded."]}

[cicharka]

Just FYI, this PR should fix such issue that I met recently.

TASK [upgrade : k8s/install | Install kubeadm 1.20.12 package for Debian family] *************************************************************************
fatal: [ec2-18-212-2-237.compute-1.amazonaws.com]: FAILED! => {"cache_update_time": 1639014589, "cache_updated": true, "changed": false, "msg": "'/usr/bin/apt-get -y -o \"Dpkg::Options::=--force-confdef\" -o \"Dpkg::Options::=--force-confold\"      install 'kubeadm=1.20.12-00' 'kubelet=1.20.12-00'' failed: E: Packages were downgraded and -y was used without --allow-downgrades.\n", "rc": 100, "stderr": "E: Packages were downgraded and -y was used without --allow-downgrades.\n", "stderr_lines": ["E: Packages were downgraded and -y was used without --allow-downgrades."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages will be DOWNGRADED:\n  kubeadm kubelet\n0 upgraded, 0 newly installed, 2 downgraded, 0 to remove and 56 not upgraded.\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages will be DOWNGRADED:", "  kubeadm kubelet", "0 upgraded, 0 newly installed, 2 downgraded, 0 to remove and 56 not upgraded."]}

@atsikham I have not faced issue like that before, do you remember the way to reproduce it so I will be able to verify?

[atsikham]

Just FYI, this PR should fix such issue that I met recently.

TASK [upgrade : k8s/install | Install kubeadm 1.20.12 package for Debian family] *************************************************************************
fatal: [ec2-18-212-2-237.compute-1.amazonaws.com]: FAILED! => {"cache_update_time": 1639014589, "cache_updated": true, "changed": false, "msg": "'/usr/bin/apt-get -y -o \"Dpkg::Options::=--force-confdef\" -o \"Dpkg::Options::=--force-confold\"      install 'kubeadm=1.20.12-00' 'kubelet=1.20.12-00'' failed: E: Packages were downgraded and -y was used without --allow-downgrades.\n", "rc": 100, "stderr": "E: Packages were downgraded and -y was used without --allow-downgrades.\n", "stderr_lines": ["E: Packages were downgraded and -y was used without --allow-downgrades."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages will be DOWNGRADED:\n  kubeadm kubelet\n0 upgraded, 0 newly installed, 2 downgraded, 0 to remove and 56 not upgraded.\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages will be DOWNGRADED:", "  kubeadm kubelet", "0 upgraded, 0 newly installed, 2 downgraded, 0 to remove and 56 not upgraded."]}

@atsikham I have not faced issue like that before, do you remember the way to reproduce it so I will be able to verify?

To reproduce that you need to run K8s upgrade on Ubuntu, interrupt that after the new kubeadm package is installed, and then run again.

[erzetpe]

/azp run

[erzetpe]

Tests passed on Ubuntu 18.04 LTS and Red Hat 7.8.