Skip to content

Commit

Permalink
Add CloudInit CRD and controller
Browse files Browse the repository at this point in the history
The CloudInit CRD allows for k8s-native management of Elemental
cloud-init files and their distribution to nodes within the cluster
using k8s object selectors.

harvester-node-manager will place an inotify watch on the cloud-init
directory to do just-in-time reconciliation for files that correspond to
a CloudInit CR (if the CloudInit matchSelector matches the node's labels).
  • Loading branch information
Connor Kuehl committed Sep 6, 2023
1 parent b1a0219 commit 5953a05
Show file tree
Hide file tree
Showing 324 changed files with 201,969 additions and 149 deletions.
16 changes: 12 additions & 4 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -28,14 +28,14 @@ replace (
require (
github.com/ehazlett/simplelog v0.0.0-20200226020431-d374894e92a4
github.com/godbus/dbus/v5 v5.1.0
github.com/harvester/go-common v0.0.0-20230718010724-11313421a8f5
github.com/harvester/go-common v0.0.0-20230821015621-981cf2216b4a
github.com/mudler/yip v1.1.0
github.com/pkg/errors v0.9.1 // indirect
github.com/prometheus/client_golang v1.16.0
github.com/rancher/lasso v0.0.0-20230629200414-8a54b32e6792
github.com/rancher/wrangler v1.1.1-0.20230818201331-3604a6be798d
github.com/shirou/gopsutil/v3 v3.22.7
github.com/sirupsen/logrus v1.9.2
github.com/sirupsen/logrus v1.9.3
github.com/spf13/viper v1.16.0
github.com/stretchr/testify v1.8.3
github.com/twpayne/go-vfs v1.7.2
Expand All @@ -48,6 +48,13 @@ require (
k8s.io/klog v1.0.0
)

require (
github.com/fsnotify/fsnotify v1.6.0
github.com/golang/mock v1.6.0
github.com/onsi/ginkgo/v2 v2.9.2
github.com/onsi/gomega v1.27.6
)

require (
github.com/beorn7/perks v1.0.1 // indirect
github.com/cespare/xxhash/v2 v2.2.0 // indirect
Expand All @@ -56,18 +63,19 @@ require (
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/emicklei/go-restful/v3 v3.9.0 // indirect
github.com/evanphx/json-patch v5.6.0+incompatible // indirect
github.com/fsnotify/fsnotify v1.6.0 // indirect
github.com/go-logr/logr v1.2.4 // indirect
github.com/go-ole/go-ole v1.2.6 // indirect
github.com/go-openapi/jsonpointer v0.19.6 // indirect
github.com/go-openapi/jsonreference v0.20.2 // indirect
github.com/go-openapi/swag v0.22.3 // indirect
github.com/go-task/slim-sprig v0.0.0-20230315185526-52ccab3ef572 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang/protobuf v1.5.3 // indirect
github.com/google/gnostic v0.6.10-0.20230825194252-836f55b2639b // indirect
github.com/google/gnostic-models v0.6.9-0.20230804172637-c7be7c783f49 // indirect
github.com/google/go-cmp v0.5.9 // indirect
github.com/google/gofuzz v1.2.0 // indirect
github.com/google/pprof v0.0.0-20210720184732-4bb14d4b1be1 // indirect
github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 // indirect
github.com/google/uuid v1.3.0 // indirect
github.com/hashicorp/errwrap v1.1.0 // indirect
Expand Down Expand Up @@ -106,7 +114,7 @@ require (
golang.org/x/net v0.13.0 // indirect
golang.org/x/oauth2 v0.8.0 // indirect
golang.org/x/sync v0.2.0 // indirect
golang.org/x/sys v0.10.0 // indirect
golang.org/x/sys v0.11.0 // indirect
golang.org/x/term v0.10.0 // indirect
golang.org/x/text v0.11.0 // indirect
golang.org/x/time v0.3.0 // indirect
Expand Down
16 changes: 9 additions & 7 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -698,6 +698,7 @@ github.com/go-openapi/swag v0.22.3/go.mod h1:UzaqsxGiab7freDnrUUra0MwWfN/q7tE4j+
github.com/go-pdf/fpdf v0.5.0/go.mod h1:HzcnA+A23uwogo0tp9yU+l3V+KXhiESpt1PMayhOh5M=
github.com/go-pdf/fpdf v0.6.0/go.mod h1:HzcnA+A23uwogo0tp9yU+l3V+KXhiESpt1PMayhOh5M=
github.com/go-task/slim-sprig v0.0.0-20230315185526-52ccab3ef572 h1:tfuBGBXKqDEevZMzYi5KSi8KkcZtzBcTgAUUtapy0OI=
github.com/go-task/slim-sprig v0.0.0-20230315185526-52ccab3ef572/go.mod h1:9Pwr4B2jHnOSGXyyzV8ROjYa2ojvAY6HCGYYfMoC3Ls=
github.com/goccy/go-json v0.9.11/go.mod h1:6MelG93GURQebXPDq3khkgXZkazVtN9CRI+MGFi0w8I=
github.com/godbus/dbus/v5 v5.0.4/go.mod h1:xhWf0FNVPg57R7Z0UbKHbJfkEywrmjJnf7w5xrFpKfA=
github.com/godbus/dbus/v5 v5.1.0 h1:4KLkAxT3aOY8Li4FRJe/KvhoNFFxo0m6fNuFUO8QJUk=
Expand Down Expand Up @@ -816,8 +817,8 @@ github.com/googleapis/google-cloud-go-testing v0.0.0-20200911160855-bcd43fbb19e8
github.com/grpc-ecosystem/grpc-gateway v1.16.0/go.mod h1:BDjrQk3hbvj6Nolgz8mAMFbcEtjT1g+wF4CSlocrBnw=
github.com/grpc-ecosystem/grpc-gateway/v2 v2.7.0/go.mod h1:hgWBS7lorOAVIJEQMi4ZsPv9hVvWI6+ch50m39Pf2Ks=
github.com/grpc-ecosystem/grpc-gateway/v2 v2.11.3/go.mod h1:o//XUCC/F+yRGJoPO/VU0GSB0f8Nhgmxx0VIRUvaC0w=
github.com/harvester/go-common v0.0.0-20230718010724-11313421a8f5 h1:0sdmh186yfBITeUM2ZbPrU4kuU8xKIPmkKjczJiAZR8=
github.com/harvester/go-common v0.0.0-20230718010724-11313421a8f5/go.mod h1:z8gmnLv5YrXhueSrZZVnd1j8ZiXKEhzitq/BnyQU/pE=
github.com/harvester/go-common v0.0.0-20230821015621-981cf2216b4a h1:X1HGGi6CyAmRzleGLg4giCfUodG1hQCliSAhwU5ckvw=
github.com/harvester/go-common v0.0.0-20230821015621-981cf2216b4a/go.mod h1:z8gmnLv5YrXhueSrZZVnd1j8ZiXKEhzitq/BnyQU/pE=
github.com/hashicorp/errwrap v1.0.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4=
github.com/hashicorp/errwrap v1.1.0 h1:OxrOeh75EUXMY8TBjag2fzXGZ40LB6IKw45YeGUDY2I=
github.com/hashicorp/errwrap v1.1.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4=
Expand Down Expand Up @@ -887,9 +888,10 @@ github.com/mudler/yip v1.1.0 h1:QQLQhD5FQ7ojaP7s7dIll6pSGnwnIplL1zGMSg5lsHQ=
github.com/mudler/yip v1.1.0/go.mod h1:GIzGnY6+tP7kaNBsmtisdyuo4cgn/4y6bEOS3GZNtkY=
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA=
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
github.com/onsi/ginkgo v1.16.5 h1:8xi0RTUf59SOSfEtZMvwTvXYMzG4gV23XVHOZiXNtnE=
github.com/onsi/ginkgo/v2 v2.9.2 h1:BA2GMJOtfGAfagzYtrAlufIP0lq6QERkFmHLMLPwFSU=
github.com/onsi/ginkgo/v2 v2.9.2/go.mod h1:WHcJJG2dIlcCqVfBAwUCrJxSPFb6v4azBwgxeMeDuts=
github.com/onsi/gomega v1.27.6 h1:ENqfyGeS5AX/rlXDd/ETokDz93u0YufY1Pgxuy/PvWE=
github.com/onsi/gomega v1.27.6/go.mod h1:PIQNjfQwkP3aQAH7lf7j87O/5FiNr+ZR8+ipb+qQlhg=
github.com/pelletier/go-toml/v2 v2.0.8 h1:0ctb6s9mE31h0/lhu+J6OPmVeDxJn+kYnJc2jZR9tGQ=
github.com/pelletier/go-toml/v2 v2.0.8/go.mod h1:vuYfssBdrU2XDZ9bYydBu6t+6a6PYNcZljzZR9VXg+4=
github.com/phpdave11/gofpdf v1.4.2/go.mod h1:zpO6xFn9yxo3YLyMvW8HcKWVdbNqgIfOOp2dXMnm1mY=
Expand Down Expand Up @@ -935,8 +937,8 @@ github.com/shirou/gopsutil/v3 v3.22.7/go.mod h1:s648gW4IywYzUfE/KjXxUsqrqx/T2xO5
github.com/shurcooL/sanitized_anchor_name v1.0.0 h1:PdmoCO6wvbs+7yrJyMORt4/BmY5IYyJwS/kOiWx8mHo=
github.com/shurcooL/sanitized_anchor_name v1.0.0/go.mod h1:1NzhyTcUVG4SuEtjjoZeVRXNmyL/1OwPU0+IJeTBvfc=
github.com/sirupsen/logrus v1.4.2/go.mod h1:tLMulIdttU9McNUspp0xgXVQah82FyeX6MwdIuYE2rE=
github.com/sirupsen/logrus v1.9.2 h1:oxx1eChJGI6Uks2ZC4W1zpLlVgqB8ner4EuQwV4Ik1Y=
github.com/sirupsen/logrus v1.9.2/go.mod h1:naHLuLoDiP4jHNo9R0sCBMtWGeIprob74mVsIT4qYEQ=
github.com/sirupsen/logrus v1.9.3 h1:dueUQJ1C2q9oE3F7wvmSGAaVtTmUizReu6fjN8uqzbQ=
github.com/sirupsen/logrus v1.9.3/go.mod h1:naHLuLoDiP4jHNo9R0sCBMtWGeIprob74mVsIT4qYEQ=
github.com/spaolacci/murmur3 v0.0.0-20180118202830-f09979ecbc72/go.mod h1:JwIasOWyU6f++ZhiEuf87xNszmSA2myDM2Kzu9HwQUA=
github.com/spf13/afero v1.3.3/go.mod h1:5KUK8ByomD5Ti5Artl0RtHeI5pTF7MIDuXL3yY520V4=
github.com/spf13/afero v1.6.0/go.mod h1:Ai8FlHk4v/PARR026UzYexafAt9roJ7LcLMAmO6Z93I=
Expand Down Expand Up @@ -1195,8 +1197,8 @@ golang.org/x/sys v0.2.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.4.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.10.0 h1:SqMFp9UcQJZa+pmYuAKjd9xq1f0j5rLcDIk0mj4qAsA=
golang.org/x/sys v0.10.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.11.0 h1:eG7RXZHdqOJ1i+0lgLgCpSXAp6M3LYlAo6osgSi0xOM=
golang.org/x/sys v0.11.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k=
golang.org/x/term v0.10.0 h1:3R7pNqamzBraeqj/Tj8qt1aQ2HpmlC+Cx/qL/7hn4/c=
Expand Down
7 changes: 6 additions & 1 deletion main.go
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ import (
"k8s.io/client-go/tools/clientcmd"
"k8s.io/klog"

"github.com/harvester/node-manager/pkg/controller/cloudinit"
"github.com/harvester/node-manager/pkg/controller/ksmtuned"
"github.com/harvester/node-manager/pkg/controller/nodeconfig"
ctlnodeharvester "github.com/harvester/node-manager/pkg/generated/controllers/node.harvesterhci.io"
Expand Down Expand Up @@ -158,6 +159,7 @@ func run(opt *option.Option) error {
}
nodecfg := nodectl.Node().V1beta1().NodeConfig()
nds := nodes.Core().V1().Node()
cloudinits := nodectl.Node().V1beta1().CloudInit()

var ksmtunedController *ksmtuned.Controller
run := func(ctx context.Context) {
Expand All @@ -181,13 +183,16 @@ func run(opt *option.Option) error {
logrus.Fatalf("failed to register ksmtuned controller: %s", err)
}

cloudinit.Register(ctx, opt.NodeName, cloudinits, nds.Cache())

if err := start.All(ctx, opt.Threadiness, nodectl, nodes); err != nil {
logrus.Fatalf("error starting, %s", err.Error())
}

}

// start monitoring
monitorTemplate := monitor.NewMonitorTemplate(ctx, mtx, nodecfg, nds, opt.NodeName)
monitorTemplate := monitor.NewMonitorTemplate(ctx, mtx, nodecfg, nds, cloudinits, opt.NodeName)
monitorNnumbers := len(utils.GetToMonitorServices())

monitorModules := make([]interface{}, 0, monitorNnumbers)
Expand Down
62 changes: 62 additions & 0 deletions manifests/crds/node.harvesterhci.io_cloudinits.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@

---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
annotations:
{}
creationTimestamp: null
name: cloudinits.node.harvesterhci.io
spec:
group: node.harvesterhci.io
names:
kind: CloudInit
listKind: CloudInitList
plural: cloudinits
shortNames:
- nci
singular: cloudinit
scope: Cluster
versions:
- name: v1beta1
schema:
openAPIV3Schema:
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this representation
of an object. Servers should convert recognized schemas to the latest
internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type: string
kind:
description: 'Kind is a string value representing the REST resource this
object represents. Servers may infer this from the endpoint the client
submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type: string
metadata:
type: object
spec:
properties:
contents:
type: string
filename:
type: string
matchSelector:
additionalProperties:
type: string
type: object
required:
- contents
- filename
- matchSelector
type: object
required:
- spec
type: object
served: true
storage: true
status:
acceptedNames:
kind: ""
plural: ""
conditions: []
storedVersions: []
22 changes: 22 additions & 0 deletions pkg/apis/node.harvesterhci.io/v1beta1/cloudinit.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
package v1beta1

import (
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// +genclient
// +genclient:nonNamespaced
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
// +kubebuilder:resource:shortName=nci,scope=Cluster

type CloudInit struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata,omitempty"`
Spec CloudInitSpec `json:"spec"`
}

type CloudInitSpec struct {
MatchSelector map[string]string `json:"matchSelector"`
Filename string `json:"filename"`
Contents string `json:"contents"`
}
83 changes: 83 additions & 0 deletions pkg/apis/node.harvesterhci.io/v1beta1/zz_generated_deepcopy.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

17 changes: 17 additions & 0 deletions pkg/apis/node.harvesterhci.io/v1beta1/zz_generated_list_types.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading

0 comments on commit 5953a05

Please sign in to comment.