fix: Removed spurious error logging when scaling down nodes by deleting a provisioner

[ellistarn]

Fixes #

Description

• Removed spurious error logging when scaling down nodes by deleting a provisioner
• Will only fall back to the instancetype if the node is not initialized. This prevents not found errors since nodes have already initialized by the time they're scaling down. If a node was never initialized, these logs may still occur, but this is quite rare
• Decoupled instance type and provisioner from the node object, since they were not needed after instantiation logic

Before

-192-168-131-232.us-west-2.compute.internal", "namespace": "", "error": "getting providerRef, AWSNodeTemplate.karpenter.k8s.aws \"skullagate-1-myhb5tu32d\" not found"}
karpenter-5f78bb8cf8-rsq2m controller 2022-07-26T23:14:26.450Z	ERROR	controller.controller.node-state	Reconciler error	{"commit": "d43c9b2", "reconciler group": "", "reconciler kind": "Node", "name": "ip-192-168-131-232.us-west-2.compute.internal", "namespace": "", "error": "getting providerRef, AWSNodeTemplate.karpenter.k8s.aws \"skullagate-1-myhb5tu32d\" not found"}
karpenter-5f78bb8cf8-rsq2m controller 2022-07-26T23:14:26.483Z	INFO	controller.termination	Deleted node	{"commit": "d43c9b2", "node": "ip-192-168-157-192.us-west-2.compute.internal"}
karpenter-5f78bb8cf8-rsq2m controller 2022-07-26T23:14:26.555Z	ERROR	controller.controller.node-state	Reconciler error	{"commit": "d43c9b2", "reconciler group": "", "reconciler kind": "Node", "name": "ip-192-168-147-136.us-west-2.compute.internal", "namespace": "", "error": "getting providerRef, AWSNodeTemplate.karpenter.k8s.aws \"skullagate-1-myhb5tu32d\" not found"}
karpenter-5f78bb8cf8-rsq2m controller 2022-07-26T23:14:26.563Z	ERROR	controller.controller.node-state	Reconciler error	{"commit": "d43c9b2", "reconciler group": "", "reconciler kind": "Node", "name": "ip-192-168-147-136.us-west-2.compute.internal", "namespace": "", "error": "getting providerRef, AWSNodeTemplate.karpenter.k8s.aws \"skullagate-1-myhb5tu32d\" not found"}
karpenter-5f78bb8cf8-rsq2m controller 2022-07-26T23:14:26.589Z	INFO	controller.termination	Deleted node	{"commit": "d43c9b2", "node": "ip-192-168-131-232.us-west-2.compute.internal"}
karpenter-5f78bb8cf8-rsq2m controller 2022-07-26T23:14:26.605Z	ERROR	controller.controller.node-state	Reconciler error	{"commit": "d43c9b2", "reconciler group": "", "reconciler kind": "Node", "name": "ip-192-168-140-239.us-west-2.compute.internal", "namespace": "", "error": "getting providerRef, AWSNodeTemplate.karpenter.k8s.aws \"skullagate-1-myhb5tu32d\" not found"}
karpenter-5f78bb8cf8-rsq2m controller 2022-07-26T23:14:26.680Z	ERROR	controller.controller.node-state	Reconciler error	{"commit": "d43c9b2", "reconciler group": "", "reconciler kind": "Node", "name": "ip-192-168-140-239.us-west-2.compute.internal", "namespace": "", "error": "getting providerRef, AWSNodeTemplate.karpenter.k8s.aws \"skullagate-1-myhb5tu32d\" not found"}

After

Scheduling Conformance
/Users/etarn/workspaces/go/src/github.com/aws/karpenter/test/suites/integration/scheduling_test.go:15
  should provision a node for a deployment
  /Users/etarn/workspaces/go/src/github.com/aws/karpenter/test/suites/integration/scheduling_test.go:28
------------------------------
    logger.go:130: 2022-07-26T16:54:27.133-0700	INFO	environment/expectations.go:163	2022-07-26T23:52:46.067Z	INFO	controller.provisioning	Found 2 provisionable pod(s)	{"commit": "603e0cc"}
        2022-07-26T23:52:46.067Z	INFO	controller.provisioning	Computed 1 new node(s) will fit 2 pod(s)	{"commit": "603e0cc"}
        2022-07-26T23:52:46.272Z	DEBUG	controller.provisioning.cloudprovider	Discovered subnets: [subnet-0b8b72961ac5f6ae5 (us-west-2c) subnet-04f48425372a1a1d4 (us-west-2b) subnet-0c83fc4dfc9db0fcd (us-west-2d) subnet-0a6370e51abc297ec (us-west-2c) subnet-0d2208da7db347297 (us-west-2d) subnet-0ba03e77da1e04b99 (us-west-2b)]	{"commit": "603e0cc", "provisioner": "parrotshade-33-3zajj3qzap"}
        2022-07-26T23:52:46.392Z	DEBUG	controller.provisioning.cloudprovider	Discovered security groups: [sg-015bab08a555f34b6 sg-04a8654e84157eb83]	{"commit": "603e0cc", "provisioner": "parrotshade-33-3zajj3qzap"}
        2022-07-26T23:52:46.394Z	DEBUG	controller.provisioning.cloudprovider	Discovered kubernetes version 1.21	{"commit": "603e0cc", "provisioner": "parrotshade-33-3zajj3qzap"}
        2022-07-26T23:52:46.428Z	DEBUG	controller.provisioning.cloudprovider	Discovered ami-00cf63b12c53803a5 for query "/aws/service/eks/optimized-ami/1.21/amazon-linux-2/recommended/image_id"	{"commit": "603e0cc", "provisioner": "parrotshade-33-3zajj3qzap"}
        2022-07-26T23:52:46.605Z	DEBUG	controller.provisioning.cloudprovider	Created launch template, Karpenter-test-17701926932498654958	{"commit": "603e0cc", "provisioner": "parrotshade-33-3zajj3qzap"}
        2022-07-26T23:52:49.729Z	INFO	controller.provisioning.cloudprovider	Launched instance: i-0dc7af21f03cdea6c, hostname: ip-192-168-130-131.us-west-2.compute.internal, type: t3a.micro, zone: us-west-2c, capacityType: on-demand	{"commit": "603e0cc", "provisioner": "parrotshade-33-3zajj3qzap"}
        2022-07-26T23:52:49.758Z	INFO	controller.provisioning	Created node with 2 pods requesting {"cpu":"125m","pods":"4"} from types t3a.micro, t3.micro, t3a.small, t3.small, t3a.medium and 388 other(s)	{"commit": "603e0cc", "provisioner": "parrotshade-33-3zajj3qzap"}
        2022-07-26T23:52:49.758Z	INFO	controller.provisioning	Waiting for unschedulable pods	{"commit": "603e0cc"}
        2022-07-26T23:52:49.759Z	DEBUG	controller.events	Normal	{"commit": "603e0cc", "object": {"kind":"Pod","namespace":"default","name":"bunnyquiver-35-miqnbxblbw-6f886d6ddf-kb7qt","uid":"546e5251-e7a7-44c8-aa85-d85be9d24fea","apiVersion":"v1","resourceVersion":"2840024"}, "reason": "NominatePod", "message": "Pod should schedule on ip-192-168-130-131.us-west-2.compute.internal"}
        2022-07-26T23:52:49.759Z	DEBUG	controller.events	Normal	{"commit": "603e0cc", "object": {"kind":"Pod","namespace":"default","name":"bunnyquiver-35-miqnbxblbw-6f886d6ddf-sp24s","uid":"744ffbef-c368-477e-9635-07113fe44725","apiVersion":"v1","resourceVersion":"2840028"}, "reason": "NominatePod", "message": "Pod should schedule on ip-192-168-130-131.us-west-2.compute.internal"}
        2022-07-26T23:52:51.739Z	DEBUG	controller.provisioning	Discovered 550 EC2 instance types	{"commit": "603e0cc"}
        2022-07-26T23:52:51.880Z	DEBUG	controller.provisioning	Discovered EC2 instance types zonal offerings	{"commit": "603e0cc"}
        2022-07-26T23:54:12.386Z	INFO	controller.termination	Cordoned node	{"commit": "603e0cc", "node": "ip-192-168-130-131.us-west-2.compute.internal"}
        2022-07-26T23:54:25.930Z	INFO	controller.termination	Deleted node	{"commit": "603e0cc", "node": "ip-192-168-130-131.us-west-2.compute.internal"}

How was this change tested?

• make test

Does this change impact docs?

• Yes, PR includes docs updates
• Yes, issue opened: #
• No

Release Note

None

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[netlify]

✅ Deploy Preview for karpenter-docs-prod canceled.

Name	Link
🔨 Latest commit	0db7f5c
🔍 Latest deploy log	https://app.netlify.com/sites/karpenter-docs-prod/deploys/62e1b5db51150a0008685825