Add chrony for time sync for Azure image #240

nader-ziada · 2020-05-21T16:08:00Z

Chrony is the only way to sync time from host using PTP source according to the azure team

Related to issue 539 in cluster-api-provider-azure

nader-ziada · 2020-05-21T16:08:22Z

detiber · 2020-05-21T16:22:01Z

images/capi/packer/azure/packer.json

+    {
+      "type": "shell",
+      "inline": [
+        "sudo apt-get -qq update && sudo apt-get -qqy install chrony &&  sudo systemctl restart chrony.service"


Could this be moved to the Ansible code instead? I suspect there will also need to be some additional modifications to the chrony config to configure the ptp time source, as described in https://docs.microsoft.com/en-us/azure/virtual-machines/linux/time-sync#chrony.

ok, will look into how to do that using ansible instead

also, if there was a daemon already present, need to make sure that's disabled and removed.

The ubuntu bionic docs say to install it https://ubuntu.com/blog/ubuntu-bionic-using-chrony-to-configure-ntp but I will double check

CecileRobertMichon · 2020-05-21T22:14:45Z

@nader-ziada how do you plan on testing this change? We should at least make sure the result is what we want before changing image builder (ie. building an image with these changes and using that to build a CAPZ cluster).

randomvariable · 2020-05-21T22:38:17Z

images/capi/ansible/roles/providers/tasks/azure.yml

+- name: fix chrony drift
+  shell: chronyc makestep 1.0 -1
+  args:
+    executable: /bin/bash


This step isn't relevant at the point of creating an image, as new machines are going to be created anyway with a completely different time.

Does chronyc actually change the config, or just the current process config?

Just checked locally on Fedora 32:

cluster-api on  master [⇡$] at ☸️ kind-kind took 5s ➜ sudo chronyc makestep 1.0 -1 200 OK cluster-api on  master [⇡$] at ☸️ kind-kind ➜ cat /etc/chrony.conf # These servers were defined in the installation: pool time.google.com iburst # Use public servers from the pool.ntp.org project. # Please consider joining the pool (http://www.pool.ntp.org/join.html). # Record the rate at which the system clock gains/losses time. driftfile /var/lib/chrony/drift # Allow the system clock to be stepped in the first three updates # if its offset is larger than 1 second. makestep 1.0 3 # Enable kernel synchronization of the real-time clock (RTC). rtcsync # Enable hardware timestamping on all interfaces that support it. #hwtimestamp * # Increase the minimum number of selectable sources required to adjust # the system clock. #minsources 2 # Allow NTP client access from local network. #allow 192.168.0.0/16 # Serve time even if not synchronized to a time source. #local stratum 10 # Specify file containing keys for NTP authentication. keyfile /etc/chrony.keys # Get TAI-UTC offset and leap seconds from the system tz database. leapsectz right/UTC # Specify directory for log files. logdir /var/log/chrony # Select which information is logged. #log measurements statistics tracking

might be actually changing the current process, i need to double check

Probably want to add something like

- name: Configure PTP lineinfile: path: /etc/chrony/chrony.conf create: yes line: refclock PHC /dev/ptp0 poll 3 dpoll -2 offset 0 - name: Ensure makestep parameter set as per Azure recommendation lineinfile: path: /etc/chrony/chrony.conf regexp: '^makestep' line: makestep 1.0 -1

Thank you, I was just trying to to figure out how to do this in ansible!

randomvariable · 2020-05-21T22:38:36Z

images/capi/ansible/roles/providers/tasks/azure.yml

+
+- name: restart chrony.service to pick up config changes
+  systemd:
+    state: restarted


Suggested change

state: restarted

enabled: yes

state: restarted

Better to be explicit that we want it enabled on boot. The restart isn't technically necessary because of image creation.

nader-ziada · 2020-05-21T22:44:30Z

@CecileRobertMichon yes, I build the image and created a vm with it manually, next will try it with a cluster

CecileRobertMichon · 2020-05-21T23:44:02Z

/cc @jackfrancis

(related to Azure/aks-engine#2552)

k8s-ci-robot · 2020-05-21T23:44:03Z

@CecileRobertMichon: GitHub didn't allow me to request PR reviews from the following users: jackfrancis.

Note that only kubernetes-sigs members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

/cc @jackfrancis

(related to Azure/aks-engine#2552)

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

nader-ziada · 2020-05-21T23:44:27Z

created the image and VM from it and this is what I see:

$ chronyc tracking
Reference ID    : 50484330 (PHC0)
Stratum         : 1
Ref time (UTC)  : Thu May 21 23:35:23 2020
System time     : 0.000013028 seconds slow of NTP time
Last offset     : -0.000015507 seconds
RMS offset      : 0.000255382 seconds
Frequency       : 72.325 ppm slow
Residual freq   : -0.054 ppm
Skew            : 1.491 ppm
Root delay      : 0.000000001 seconds
Root dispersion : 0.000033918 seconds
Update interval : 8.0 seconds
Leap status     : Normal

$ chronyc activity
200 OK
8 sources online
0 sources offline
0 sources doing burst (return to online)
0 sources doing burst (return to offline)
0 sources with unknown address

$ chronyc sources
210 Number of sources = 9
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
#* PHC0                          0   3   377    12    +85us[  +98us] +/-   17us
^- alphyn.canonical.com          2   6   357    22   +532us[ +548us] +/-   68ms
^- golem.canonical.com           2   6   377    16   +816us[ +834us] +/-   69ms
^- chilipepper.canonical.com     2   6   377    18   +851us[ +868us] +/-   73ms
^- pugot.canonical.com           2   6   377    19   +685us[ +702us] +/-   82ms
^- atl0.fairy.mattnordhoff.>     2   6   377    21   +542us[ +559us] +/-   29ms
^- europa.netbunker.org          2   6   377    18  +1980us[+1997us] +/-   29ms
^- t1.time.bf1.yahoo.com         2   6   377    19  +1042us[+1059us] +/-   46ms
^- 198.255.68.106                2   6   377    19  +1253us[+1270us] +/-   80ms

~$ cat /etc/chrony/chrony.conf
# Welcome to the chrony configuration file. See chrony.conf(5) for more
# information about usuable directives.

# This will use (up to):
# - 4 sources from ntp.ubuntu.com which some are ipv6 enabled
# - 2 sources from 2.ubuntu.pool.ntp.org which is ipv6 enabled as well
# - 1 source from [01].ubuntu.pool.ntp.org each (ipv4 only atm)
# This means by default, up to 6 dual-stack and up to 2 additional IPv4-only
# sources will be used.
# At the same time it retains some protection against one of the entries being
# down (compare to just using one of the lines). See (LP: #1754358) for the
# discussion.
#
# About using servers from the NTP Pool Project in general see (LP: #104525).
# Approved by Ubuntu Technical Board on 2011-02-08.
# See http://www.pool.ntp.org/join.html for more information.
pool ntp.ubuntu.com        iburst maxsources 4
pool 0.ubuntu.pool.ntp.org iburst maxsources 1
pool 1.ubuntu.pool.ntp.org iburst maxsources 1
pool 2.ubuntu.pool.ntp.org iburst maxsources 2

# This directive specify the location of the file containing ID/key pairs for
# NTP authentication.
keyfile /etc/chrony/chrony.keys

# This directive specify the file into which chronyd will store the rate
# information.
driftfile /var/lib/chrony/chrony.drift

# Uncomment the following line to turn logging on.
#log tracking measurements statistics

# Log files location.
logdir /var/log/chrony

# Stop bad estimates upsetting machine clock.
maxupdateskew 100.0

# This directive enables kernel synchronisation (every 11 minutes) of the
# real-time clock. Note that it can’t be used along with the 'rtcfile' directive.
rtcsync

# Step the system clock instead of slewing it if the adjustment is larger than
# one second, but only in the first three clock updates.
makestep 1.0 -1

refclock PHC /dev/ptp0 poll 3 dpoll -2 offset 0

nader-ziada · 2020-05-22T14:59:54Z

I created a few Azure clusters using the image I generated from here and everything is working fine, not sure if there is a special test we need to check on the cluster, but at least this doesn't break anything and chrony is running as in the above comment.

nader-ziada · 2020-05-26T20:15:00Z

@CecileRobertMichon is it possible for you to try this out?

CecileRobertMichon · 2020-05-29T21:45:51Z

@detiber @randomvariable do we have plans to include E2E testing as a presubmit for image-buidler PRs? Seems like it would be useful in cases like this

CecileRobertMichon · 2020-05-29T21:47:40Z

@nader-ziada this looks good to me, but we really need a way to be able to test changes to the image in a non-manual way in the future :)

/lgtm

nader-ziada · 2020-05-29T22:59:12Z

@CecileRobertMichon i can work on tests but better if we can discuss what scope to cover

nader-ziada · 2020-06-01T14:06:09Z

@detiber ready for another look, Thanks

detiber · 2020-06-01T17:51:32Z

/cc @codenrhoden to speak to testing plans for image-builder

k8s-ci-robot · 2020-06-01T17:51:35Z

@detiber: GitHub didn't allow me to request PR reviews from the following users: image-builder, to, speak, testing, for.

Note that only kubernetes-sigs members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

/cc @codenrhoden to speak to testing plans for image-builder

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

detiber · 2020-06-01T17:55:54Z

images/capi/ansible/roles/providers/tasks/azure.yml

+- name: Ansible apt install chrony
+  apt:
+    name: chrony
+    state: present


This should probably be wrapped in a conditional to only run when the OS is Ubuntu similar to:

image-builder/images/capi/ansible/roles/providers/tasks/aws.yml

Lines 38 to 40 in c680d2c

- name: install aws agents Ubuntu

shell: snap install amazon-ssm-agent --classic

when: ansible_distribution == "Ubuntu"

Relatively minor nit, but would have an impact if/when other OS images are supported for Azure images.

updated to add check for Ubuntu

Chrony is the only way to sync time from host using PTP source according to the azure team

detiber · 2020-06-01T18:59:46Z

/lgtm
/assign @codenrhoden

nader-ziada · 2020-06-08T13:24:00Z

Hi @codenrhoden , do you have any concerns with this change?

codenrhoden · 2020-06-15T19:03:50Z

/approve

Thanks for this @nader-ziada! And for trying out clusters with CAPZ first since there are no blocking PR tests still...

@CecileRobertMichon:

do we have plans to include E2E testing as a presubmit for image-buidler PRs? Seems like it would be useful in cases like this

Short answer is "yes". 😆 But definitely lacking on details so far. But I am in 100% agreement that this is necessary. It's in my near-term plans to start to focus on this.

k8s-ci-robot · 2020-06-15T19:04:10Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: codenrhoden, nader-ziada

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~images/capi/OWNERS~~ [codenrhoden]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels May 21, 2020

k8s-ci-robot requested review from davidewatson and vincepri May 21, 2020 16:08

k8s-ci-robot requested a review from CecileRobertMichon May 21, 2020 16:08

nader-ziada mentioned this pull request May 21, 2020

Proposal: use host based time sync instead of systemd-timesyncd by default kubernetes-sigs/cluster-api-provider-azure#539

Closed

detiber reviewed May 21, 2020

View reviewed changes

nader-ziada force-pushed the azure-chrony branch from f28b10e to d80932e Compare May 21, 2020 22:26

k8s-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels May 21, 2020

randomvariable reviewed May 21, 2020

View reviewed changes

nader-ziada force-pushed the azure-chrony branch from d80932e to 19ba437 Compare May 21, 2020 23:41

nader-ziada force-pushed the azure-chrony branch from 19ba437 to 39a5c8c Compare May 21, 2020 23:47

k8s-ci-robot assigned CecileRobertMichon May 29, 2020

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 29, 2020

k8s-ci-robot requested a review from codenrhoden June 1, 2020 17:51

detiber reviewed Jun 1, 2020

View reviewed changes

Add chrony for time sync for Azure image

58794c3

Chrony is the only way to sync time from host using PTP source according to the azure team

nader-ziada force-pushed the azure-chrony branch from 39a5c8c to 58794c3 Compare June 1, 2020 18:49

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 1, 2020

k8s-ci-robot assigned codenrhoden and detiber Jun 1, 2020

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 1, 2020

nader-ziada mentioned this pull request Jun 11, 2020

[e2e] cluster should have healthy time synchronization kubernetes-sigs/cluster-api-provider-azure#705

Closed

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 15, 2020

k8s-ci-robot merged commit bfcc794 into kubernetes-sigs:master Jun 15, 2020

mboersma mentioned this pull request Jul 17, 2020

Update Kubernetes versions in e2e test and docs kubernetes-sigs/cluster-api-provider-azure#800

Merged

3 tasks

shysank mentioned this pull request Feb 18, 2022

don’t validate timesync during capz E2E kubernetes-sigs/cluster-api-provider-azure#2099

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add chrony for time sync for Azure image #240

Add chrony for time sync for Azure image #240

nader-ziada commented May 21, 2020

nader-ziada commented May 21, 2020

detiber May 21, 2020

nader-ziada May 21, 2020

randomvariable May 21, 2020

nader-ziada May 21, 2020

CecileRobertMichon commented May 21, 2020 •

edited

Loading

randomvariable May 21, 2020

randomvariable May 21, 2020

randomvariable May 21, 2020 •

edited

Loading

nader-ziada May 21, 2020

randomvariable May 21, 2020

nader-ziada May 21, 2020

randomvariable May 21, 2020 •

edited

Loading

nader-ziada commented May 21, 2020

CecileRobertMichon commented May 21, 2020

k8s-ci-robot commented May 21, 2020

nader-ziada commented May 21, 2020

nader-ziada commented May 22, 2020

nader-ziada commented May 26, 2020

CecileRobertMichon commented May 29, 2020

CecileRobertMichon commented May 29, 2020

nader-ziada commented May 29, 2020

nader-ziada commented Jun 1, 2020

detiber commented Jun 1, 2020

k8s-ci-robot commented Jun 1, 2020

detiber Jun 1, 2020

nader-ziada Jun 1, 2020

detiber commented Jun 1, 2020

nader-ziada commented Jun 8, 2020

codenrhoden commented Jun 15, 2020

k8s-ci-robot commented Jun 15, 2020

	- name: install aws agents Ubuntu
	shell: snap install amazon-ssm-agent --classic
	when: ansible_distribution == "Ubuntu"

Add chrony for time sync for Azure image #240

Add chrony for time sync for Azure image #240

Conversation

nader-ziada commented May 21, 2020

nader-ziada commented May 21, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CecileRobertMichon commented May 21, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

randomvariable May 21, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

randomvariable May 21, 2020 • edited Loading

Choose a reason for hiding this comment

nader-ziada commented May 21, 2020

CecileRobertMichon commented May 21, 2020

k8s-ci-robot commented May 21, 2020

nader-ziada commented May 21, 2020

nader-ziada commented May 22, 2020

nader-ziada commented May 26, 2020

CecileRobertMichon commented May 29, 2020

CecileRobertMichon commented May 29, 2020

nader-ziada commented May 29, 2020

nader-ziada commented Jun 1, 2020

detiber commented Jun 1, 2020

k8s-ci-robot commented Jun 1, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

detiber commented Jun 1, 2020

nader-ziada commented Jun 8, 2020

codenrhoden commented Jun 15, 2020

k8s-ci-robot commented Jun 15, 2020

CecileRobertMichon commented May 21, 2020 •

edited

Loading

randomvariable May 21, 2020 •

edited

Loading

randomvariable May 21, 2020 •

edited

Loading