Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Resource client patch for apply status #517

Merged
merged 42 commits into from
Sep 14, 2022
Merged
Changes from 39 commits
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
2016348
Add patch to resource client to cut api calls in half
Sep 8, 2022
b3d9c49
Move to apply status since consul api doesn't have patch
Sep 9, 2022
ed3ded9
Implement the other apply status functions
Sep 9, 2022
932d53b
Get rid of debug info
Sep 9, 2022
80eb2ce
Clean up extra code
Sep 9, 2022
d58ce2c
Actually use the bytes from the provided status
Sep 9, 2022
791427f
Add changelog
Sep 9, 2022
e5f2e19
Pod resource client to implement interface
Sep 9, 2022
ac38ae7
Api client to implement interface
Sep 9, 2022
63a18d3
Namespace resource client to implement interface
Sep 9, 2022
26b02e8
Use more performant one with standard k8s clients
Sep 9, 2022
a62d491
Namespace client needs too
Sep 9, 2022
76c3a35
Codegen
Sep 9, 2022
245e753
Fix reporter unit test
Sep 9, 2022
fe923b5
Fix reconciler unit tests
Sep 9, 2022
3762a69
Codegen
Sep 9, 2022
e3d5cf9
Update interface to use status client
Sep 12, 2022
3eac75a
Codegen
Sep 12, 2022
0f5cca9
Use namespaced status by default
Sep 12, 2022
5bb6c80
Fix reporter unit test
Sep 12, 2022
d911353
Fix reconciler unit test
Sep 12, 2022
7b0c2a4
Prefer merge patch type
Sep 12, 2022
16fbc92
Ensure we render all fields
Sep 12, 2022
dbfe07d
Codegen
Sep 12, 2022
770de5d
Go mod tidy
Sep 12, 2022
9ebc5c0
Merge refs/heads/master into resource_client_patch
soloio-bulldozer[bot] Sep 13, 2022
c0bf2f8
Prefer json patch type
Sep 13, 2022
f5c5bbe
Merge branch 'resource_client_patch' of github.com:solo-io/solo-kit i…
Sep 13, 2022
965798f
Ensure statuses is always defaulted
Sep 13, 2022
26eeb55
Prefer json encoding
Sep 13, 2022
b6bf50a
Codegen
Sep 13, 2022
b2896a8
Improve comment readability
Sep 13, 2022
0c2cbec
Get rid of gogo
Sep 13, 2022
7de9cec
Get rid of gogo
Sep 13, 2022
ce6d2ed
Get rid of gogo, with codegen
Sep 13, 2022
dd12a82
Move shared impl to shared package
Sep 13, 2022
9b56c85
Move shared impl for jsonpatch to shared package
Sep 13, 2022
48d1f46
Move other clients to shared impl for jsonpatch
Sep 13, 2022
80b78a0
Remove gogo
Sep 13, 2022
7ed5390
Only clone if we are going to write status
Sep 14, 2022
b9532a5
Be clear we are patching the status
Sep 14, 2022
fa41615
Add note in changelog about rendering enums as strings
Sep 14, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions changelog/v0.30.3/patch-status.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
changelog:
- type: FIX
resolvesIssue: false
description: Improve scalability of v2 status reporter by using k8s patch instead of read then write.
issueLink: https://github.com/solo-io/gloo/issues/7076
Original file line number Diff line number Diff line change
@@ -8,6 +8,7 @@ import (

"github.com/solo-io/solo-kit/api/external/kubernetes/customresourcedefinition"

"github.com/solo-io/solo-kit/pkg/api/shared"
"github.com/solo-io/solo-kit/pkg/api/v1/clients"
"github.com/solo-io/solo-kit/pkg/api/v1/clients/common"
"github.com/solo-io/solo-kit/pkg/api/v1/resources"
@@ -18,6 +19,7 @@ import (
apierrors "k8s.io/apimachinery/pkg/api/errors"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/labels"
"k8s.io/apimachinery/pkg/types"
)

type customResourceDefinitionResourceClient struct {
@@ -124,6 +126,34 @@ func (rc *customResourceDefinitionResourceClient) Write(resource resources.Resou
return rc.Read(customResourceDefinitionObj.Namespace, customResourceDefinitionObj.Name, clients.ReadOpts{Ctx: opts.Ctx})
}

func (rc *customResourceDefinitionResourceClient) ApplyStatus(statusClient resources.StatusClient, inputResource resources.InputResource, opts clients.ApplyStatusOpts) (resources.Resource, error) {
name := inputResource.GetMetadata().GetName()
namespace := inputResource.GetMetadata().GetNamespace()
if err := resources.ValidateName(name); err != nil {
return nil, errors.Wrapf(err, "validation error")
}
opts = opts.WithDefaults()

data, err := shared.GetJsonPatchData(inputResource)
if err != nil {
return nil, errors.Wrapf(err, "error getting status json patch data")
}

customResourceDefinitionObj, err := rc.apiExts.ApiextensionsV1().CustomResourceDefinitions().Patch(opts.Ctx, name, types.JSONPatchType, data, metav1.PatchOptions{})
if err != nil {
if apierrors.IsNotFound(err) {
return nil, errors.NewNotExistErr(namespace, name, err)
}
return nil, errors.Wrapf(err, "patching customResourceDefinitionObj from kubernetes")
}
resource := FromKubeCustomResourceDefinition(customResourceDefinitionObj)

if resource == nil {
return nil, errors.Errorf("customResourceDefinitionObj %v is not kind %v", name, rc.Kind())
}
return resource, nil
}

func (rc *customResourceDefinitionResourceClient) Delete(namespace, name string, opts clients.DeleteOpts) error {
opts = opts.WithDefaults()
if !rc.exist(opts.Ctx, namespace, name) {
30 changes: 30 additions & 0 deletions pkg/api/external/kubernetes/deployment/resource_client.go
Original file line number Diff line number Diff line change
@@ -5,6 +5,7 @@ import (
"sort"

kubedeployment "github.com/solo-io/solo-kit/api/external/kubernetes/deployment"
"github.com/solo-io/solo-kit/pkg/api/shared"
"github.com/solo-io/solo-kit/pkg/api/v1/clients/common"
"github.com/solo-io/solo-kit/pkg/api/v1/clients/kube/cache"
skkube "github.com/solo-io/solo-kit/pkg/api/v1/resources/common/kubernetes"
@@ -16,6 +17,7 @@ import (
apierrors "k8s.io/apimachinery/pkg/api/errors"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/labels"
"k8s.io/apimachinery/pkg/types"
"k8s.io/client-go/kubernetes"
)

@@ -123,6 +125,34 @@ func (rc *deploymentResourceClient) Write(resource resources.Resource, opts clie
return rc.Read(deploymentObj.Namespace, deploymentObj.Name, clients.ReadOpts{Ctx: opts.Ctx})
}

func (rc *deploymentResourceClient) ApplyStatus(statusClient resources.StatusClient, inputResource resources.InputResource, opts clients.ApplyStatusOpts) (resources.Resource, error) {
name := inputResource.GetMetadata().GetName()
namespace := inputResource.GetMetadata().GetNamespace()
if err := resources.ValidateName(name); err != nil {
return nil, errors.Wrapf(err, "validation error")
}
opts = opts.WithDefaults()

data, err := shared.GetJsonPatchData(inputResource)
if err != nil {
return nil, errors.Wrapf(err, "error getting status json patch data")
}

deploymentObj, err := rc.Kube.AppsV1().Deployments(namespace).Patch(opts.Ctx, name, types.JSONPatchType, data, metav1.PatchOptions{})
if err != nil {
if apierrors.IsNotFound(err) {
return nil, errors.NewNotExistErr(namespace, name, err)
}
return nil, errors.Wrapf(err, "patching deployment from kubernetes")
}
resource := FromKubeDeployment(deploymentObj)

if resource == nil {
return nil, errors.Errorf("deployment %v is not kind %v", name, rc.Kind())
}
return resource, nil
}

func (rc *deploymentResourceClient) Delete(namespace, name string, opts clients.DeleteOpts) error {
opts = opts.WithDefaults()
if !rc.exist(opts.Ctx, namespace, name) {
29 changes: 29 additions & 0 deletions pkg/api/external/kubernetes/job/resource_client.go
Original file line number Diff line number Diff line change
@@ -5,6 +5,7 @@ import (
"sort"

kubejob "github.com/solo-io/solo-kit/api/external/kubernetes/job"
"github.com/solo-io/solo-kit/pkg/api/shared"
"github.com/solo-io/solo-kit/pkg/api/v1/clients/common"
"github.com/solo-io/solo-kit/pkg/api/v1/clients/kube/cache"
skkube "github.com/solo-io/solo-kit/pkg/api/v1/resources/common/kubernetes"
@@ -16,6 +17,7 @@ import (
apierrors "k8s.io/apimachinery/pkg/api/errors"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/labels"
"k8s.io/apimachinery/pkg/types"
"k8s.io/client-go/kubernetes"
)

@@ -123,6 +125,33 @@ func (rc *jobResourceClient) Write(resource resources.Resource, opts clients.Wri
return rc.Read(jobObj.Namespace, jobObj.Name, clients.ReadOpts{Ctx: opts.Ctx})
}

func (rc *jobResourceClient) ApplyStatus(statusClient resources.StatusClient, inputResource resources.InputResource, opts clients.ApplyStatusOpts) (resources.Resource, error) {
name := inputResource.GetMetadata().GetName()
namespace := inputResource.GetMetadata().GetNamespace()
if err := resources.ValidateName(name); err != nil {
return nil, errors.Wrapf(err, "validation error")
}
opts = opts.WithDefaults()

data, err := shared.GetJsonPatchData(inputResource)
if err != nil {
return nil, errors.Wrapf(err, "error getting status json patch data")
}
jobObj, err := rc.Kube.BatchV1().Jobs(namespace).Patch(opts.Ctx, name, types.JSONPatchType, data, metav1.PatchOptions{})
if err != nil {
if apierrors.IsNotFound(err) {
return nil, errors.NewNotExistErr(namespace, name, err)
}
return nil, errors.Wrapf(err, "patching job from kubernetes")
}
resource := FromKubeJob(jobObj)

if resource == nil {
return nil, errors.Errorf("job %v is not kind %v", name, rc.Kind())
}
return resource, nil
}

func (rc *jobResourceClient) Delete(namespace, name string, opts clients.DeleteOpts) error {
opts = opts.WithDefaults()
if !rc.exist(opts.Ctx, namespace, name) {
29 changes: 29 additions & 0 deletions pkg/api/external/kubernetes/namespace/resource_client.go
Original file line number Diff line number Diff line change
@@ -7,6 +7,7 @@ import (
"github.com/bugsnag/bugsnag-go/errors"
"github.com/rotisserie/eris"
kubenamespace "github.com/solo-io/solo-kit/api/external/kubernetes/namespace"
"github.com/solo-io/solo-kit/pkg/api/shared"
"github.com/solo-io/solo-kit/pkg/api/v1/clients"
"github.com/solo-io/solo-kit/pkg/api/v1/clients/common"
"github.com/solo-io/solo-kit/pkg/api/v1/clients/kube/cache"
@@ -17,6 +18,7 @@ import (
apierrors "k8s.io/apimachinery/pkg/api/errors"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/labels"
"k8s.io/apimachinery/pkg/types"
"k8s.io/client-go/kubernetes"
)

@@ -122,6 +124,33 @@ func (rc *namespaceResourceClient) Write(resource resources.Resource, opts clien
return rc.Read(namespaceObj.Namespace, namespaceObj.Name, clients.ReadOpts{Ctx: opts.Ctx})
}

func (rc *namespaceResourceClient) ApplyStatus(statusClient resources.StatusClient, inputResource resources.InputResource, opts clients.ApplyStatusOpts) (resources.Resource, error) {
name := inputResource.GetMetadata().GetName()
namespace := inputResource.GetMetadata().GetNamespace()
if err := resources.ValidateName(name); err != nil {
return nil, eris.Wrapf(err, "validation error")
}
opts = opts.WithDefaults()

data, err := shared.GetJsonPatchData(inputResource)
if err != nil {
return nil, eris.Wrapf(err, "error getting status json patch data")
}
namespaceObj, err := rc.Kube.CoreV1().Namespaces().Patch(opts.Ctx, name, types.JSONPatchType, data, metav1.PatchOptions{})
if err != nil {
if apierrors.IsNotFound(err) {
return nil, skerrors.NewNotExistErr(namespace, name, err)
}
return nil, eris.Wrapf(err, "reading namespaceObj from kubernetes")
}
resource := FromKubeNamespace(namespaceObj)

if resource == nil {
return nil, eris.Errorf("namespaceObj %v is not kind %v", name, rc.Kind())
}
return resource, nil
}

func (rc *namespaceResourceClient) Delete(namespace, name string, opts clients.DeleteOpts) error {
opts = opts.WithDefaults()
if !rc.exist(opts.Ctx, namespace, name) {
29 changes: 29 additions & 0 deletions pkg/api/external/kubernetes/pod/resource_client.go
Original file line number Diff line number Diff line change
@@ -5,6 +5,7 @@ import (
"sort"

kubepod "github.com/solo-io/solo-kit/api/external/kubernetes/pod"
"github.com/solo-io/solo-kit/pkg/api/shared"
"github.com/solo-io/solo-kit/pkg/api/v1/clients/common"
"github.com/solo-io/solo-kit/pkg/api/v1/clients/kube/cache"
skkube "github.com/solo-io/solo-kit/pkg/api/v1/resources/common/kubernetes"
@@ -16,6 +17,7 @@ import (
apierrors "k8s.io/apimachinery/pkg/api/errors"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/labels"
"k8s.io/apimachinery/pkg/types"
"k8s.io/client-go/kubernetes"
)

@@ -123,6 +125,33 @@ func (rc *podResourceClient) Write(resource resources.Resource, opts clients.Wri
return rc.Read(podObj.Namespace, podObj.Name, clients.ReadOpts{Ctx: opts.Ctx})
}

func (rc *podResourceClient) ApplyStatus(statusClient resources.StatusClient, inputResource resources.InputResource, opts clients.ApplyStatusOpts) (resources.Resource, error) {
name := inputResource.GetMetadata().GetName()
namespace := inputResource.GetMetadata().GetNamespace()
if err := resources.ValidateName(name); err != nil {
return nil, errors.Wrapf(err, "validation error")
}
opts = opts.WithDefaults()

data, err := shared.GetJsonPatchData(inputResource)
if err != nil {
return nil, errors.Wrapf(err, "error getting status json patch data")
}
podObj, err := rc.Kube.CoreV1().Pods(namespace).Patch(opts.Ctx, name, types.JSONPatchType, data, metav1.PatchOptions{})
if err != nil {
if apierrors.IsNotFound(err) {
return nil, errors.NewNotExistErr(namespace, name, err)
}
return nil, errors.Wrapf(err, "patching podObj from kubernetes")
}
resource := FromKubePod(podObj)

if resource == nil {
return nil, errors.Errorf("podObj %v is not kind %v", name, rc.Kind())
}
return resource, nil
}

func (rc *podResourceClient) Delete(namespace, name string, opts clients.DeleteOpts) error {
opts = opts.WithDefaults()
if !rc.exist(opts.Ctx, namespace, name) {
29 changes: 29 additions & 0 deletions pkg/api/external/kubernetes/service/resource_client.go
Original file line number Diff line number Diff line change
@@ -5,6 +5,7 @@ import (
"sort"

kubeservice "github.com/solo-io/solo-kit/api/external/kubernetes/service"
"github.com/solo-io/solo-kit/pkg/api/shared"
"github.com/solo-io/solo-kit/pkg/api/v1/clients/common"
"github.com/solo-io/solo-kit/pkg/api/v1/clients/kube/cache"
skkube "github.com/solo-io/solo-kit/pkg/api/v1/resources/common/kubernetes"
@@ -16,6 +17,7 @@ import (
apierrors "k8s.io/apimachinery/pkg/api/errors"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/labels"
"k8s.io/apimachinery/pkg/types"
"k8s.io/client-go/kubernetes"
)

@@ -123,6 +125,33 @@ func (rc *serviceResourceClient) Write(resource resources.Resource, opts clients
return rc.Read(serviceObj.Namespace, serviceObj.Name, clients.ReadOpts{Ctx: opts.Ctx})
}

func (rc *serviceResourceClient) ApplyStatus(statusClient resources.StatusClient, inputResource resources.InputResource, opts clients.ApplyStatusOpts) (resources.Resource, error) {
name := inputResource.GetMetadata().GetName()
namespace := inputResource.GetMetadata().GetNamespace()
if err := resources.ValidateName(name); err != nil {
return nil, errors.Wrapf(err, "validation error")
}
opts = opts.WithDefaults()

data, err := shared.GetJsonPatchData(inputResource)
if err != nil {
return nil, errors.Wrapf(err, "error getting status json patch data")
}
serviceObj, err := rc.Kube.CoreV1().Services(namespace).Patch(opts.Ctx, name, types.JSONPatchType, data, metav1.PatchOptions{})
if err != nil {
if apierrors.IsNotFound(err) {
return nil, errors.NewNotExistErr(namespace, name, err)
}
return nil, errors.Wrapf(err, "patching serviceObj from kubernetes")
}
resource := FromKubeService(serviceObj)

if resource == nil {
return nil, errors.Errorf("serviceObj %v is not kind %v", name, rc.Kind())
}
return resource, nil
}

func (rc *serviceResourceClient) Delete(namespace, name string, opts clients.DeleteOpts) error {
opts = opts.WithDefaults()
if !rc.exist(opts.Ctx, namespace, name) {
71 changes: 71 additions & 0 deletions pkg/api/shared/status.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
package shared

import (
"bytes"
"fmt"

"github.com/golang/protobuf/jsonpb"
"github.com/solo-io/solo-kit/pkg/api/v1/clients"
"github.com/solo-io/solo-kit/pkg/api/v1/resources"
"github.com/solo-io/solo-kit/pkg/errors"
)

// ApplyStatus is used by clients that don't support patch updates to resource statuses (e.g. consul, files, in-memory)
func ApplyStatus(rc clients.ResourceClient, statusClient resources.StatusClient, inputResource resources.InputResource, opts clients.ApplyStatusOpts) (resources.Resource, error) {
name := inputResource.GetMetadata().GetName()
namespace := inputResource.GetMetadata().GetNamespace()
res, err := rc.Read(namespace, name, clients.ReadOpts{
Ctx: opts.Ctx,
Cluster: opts.Cluster,
})

if err != nil {
return nil, errors.Wrapf(err, "error reading before applying status")
}

inputRes, ok := res.(resources.InputResource)
if !ok {
return nil, errors.Errorf("error converting resource of type %T to input resource to apply status", res)
}

statusClient.SetStatus(inputRes, statusClient.GetStatus(inputResource))
updatedRes, err := rc.Write(inputRes, clients.WriteOpts{
Ctx: opts.Ctx,
OverwriteExisting: true,
})

if err != nil {
return nil, errors.Wrapf(err, "error writing to apply status")
}
return updatedRes, nil
}

// GetJsonPatchData returns the json patch data for the input resource.
// Prefer using json patch for single api call status updates when supported (e.g. k8s) to avoid ratelimiting
// to the k8s apiserver (e.g. https://github.com/solo-io/gloo/blob/a083522af0a4ce22f4d2adf3a02470f782d5a865/projects/gloo/api/v1/settings.proto#L337-L350)
func GetJsonPatchData(inputResource resources.InputResource) ([]byte, error) {
namespacedStatuses := inputResource.GetNamespacedStatuses().GetStatuses()
if len(namespacedStatuses) != 1 {
// we only expect our namespace to report here; we don't want to blow away statuses from other reporters
return nil, errors.Errorf("unexpected number of namespaces in input resource: %v", len(inputResource.GetNamespacedStatuses().GetStatuses()))
}
ns := ""
for loopNs := range inputResource.GetNamespacedStatuses().GetStatuses() {
ns = loopNs
}
status := inputResource.GetNamespacedStatuses().GetStatuses()[ns]

buf := &bytes.Buffer{}
var marshaller jsonpb.Marshaler
marshaller.EnumsAsInts = false // prefer jsonpb over encoding/json marshaller since it renders enum as string not int (i.e., state is human-readable)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently we write out ints for the states. If we want to switch to writing strings we should call it out in the changelog. I assume that this change won't affect any tooling around statuses because the unmarshaller will handle it the same regardless.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

happy to call that out in the changelog. yeah we currently use the jsonpb to unmarshal so it can handle either format, so im just opting for better readability

marshaller.EmitDefaults = false // keep status as small as possible
err := marshaller.Marshal(buf, status)
if err != nil {
return nil, errors.Wrapf(err, "marshalling input resource")
}

bytes := buf.Bytes()
patch := fmt.Sprintf(`[{"op": "replace", "path": "/status/statuses/%s", "value": %s}]`, ns, string(bytes)) // only replace our status so other reporters are not affected (e.g. blue-green of gloo)
data := []byte(patch)
return data, nil
}
5 changes: 5 additions & 0 deletions pkg/api/v1/clients/apiclient/resource_client.go
Original file line number Diff line number Diff line change
@@ -6,6 +6,7 @@ import (
"strings"

"github.com/golang/protobuf/ptypes"
"github.com/solo-io/solo-kit/pkg/api/shared"
"github.com/solo-io/solo-kit/pkg/api/v1/apiserver"
"github.com/solo-io/solo-kit/pkg/api/v1/clients"
"github.com/solo-io/solo-kit/pkg/api/v1/resources"
@@ -115,6 +116,10 @@ func (rc *ResourceClient) Write(resource resources.Resource, opts clients.WriteO
return written, nil
}

func (rc *ResourceClient) ApplyStatus(statusClient resources.StatusClient, inputResource resources.InputResource, opts clients.ApplyStatusOpts) (resources.Resource, error) {
return shared.ApplyStatus(rc, statusClient, inputResource, opts)
}

func (rc *ResourceClient) Delete(namespace, name string, opts clients.DeleteOpts) error {
opts = opts.WithDefaults()
opts.Ctx = metadata.AppendToOutgoingContext(opts.Ctx, "authorization", "bearer "+rc.token)
Loading