Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix scheme data race #375

Open
kdorosh opened this issue Sep 21, 2022 · 7 comments · Fixed by k8sgateway/k8sgateway#7207
Open

fix scheme data race #375

kdorosh opened this issue Sep 21, 2022 · 7 comments · Fixed by k8sgateway/k8sgateway#7207

Comments

@kdorosh
Copy link
Contributor

kdorosh commented Sep 21, 2022

e.g.

WARNING: DATA RACE
Write at 0x00c0004c1c80 by goroutine 40:
  runtime.mapassign()
      /opt/hostedtoolcache/go/1.18.2/x64/src/runtime/map.go:578 +0x0
  k8s.io/apimachinery/pkg/runtime.(*Scheme).AddKnownTypeWithName()
      /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/runtime/scheme.go:196 +0x3a4
  k8s.io/apimachinery/pkg/runtime.(*Scheme).AddKnownTypes()
      /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/runtime/scheme.go:170 +0x2e8
  sigs.k8s.io/controller-runtime/pkg/scheme.(*Builder).Register.func1()
      /home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/scheme/scheme.go:72 +0x8b
  k8s.io/apimachinery/pkg/runtime.(*SchemeBuilder).AddToScheme()
      /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/runtime/scheme_builder.go:29 +0xa1
  sigs.k8s.io/controller-runtime/pkg/scheme.(*Builder).AddToScheme()
      /home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/scheme/scheme.go:87 +0x2c
  github.com/solo-io/solo-apis/pkg/api/ratelimit.solo.io/v1alpha1.AddToScheme()
      /home/runner/go/pkg/mod/github.com/solo-io/[email protected]/pkg/api/ratelimit.solo.io/v1alpha1/register.go:32 +0x5b
  github.com/solo-io/solo-apis/pkg/api/ratelimit.solo.io/v1alpha1.NewClientsetFromConfig()
      /home/runner/go/pkg/mod/github.com/solo-io/[email protected]/pkg/api/ratelimit.solo.io/v1alpha1/clients.go:52 +0x60
  github.com/solo-io/gloo/projects/gloo/pkg/api/external/solo/ratelimit.NewRateLimitClients()
      /home/runner/work/gloo/gloo/projects/gloo/pkg/api/external/solo/ratelimit/extensions.go:54 +0xf9
  github.com/solo-io/gloo/projects/gloo/pkg/syncer/setup.RunGlooWithExtensions()
      /home/runner/work/gloo/gloo/projects/gloo/pkg/syncer/setup/setup_syncer.go:504 +0xb5a
  github.com/solo-io/gloo/projects/gloo/pkg/syncer/setup.RunGloo()
      /home/runner/work/gloo/gloo/projects/gloo/pkg/syncer/setup/setup_syncer.go:417 +0x24f
  github.com/solo-io/gloo/projects/gloo/pkg/syncer/setup.(*setupSyncer).Setup()
      /home/runner/work/gloo/gloo/projects/gloo/pkg/syncer/setup/setup_syncer.go:391 +0x21f4
  github.com/solo-io/gloo/projects/gloo/pkg/syncer/setup.(*setupSyncer).Setup-fm()
      <autogenerated>:1 +0xdc
  github.com/solo-io/gloo/test/kube2e/gloo_test.glob..func4.1.1()
      /home/runner/work/gloo/gloo/test/kube2e/gloo/setup_syncer_test.go:56 +0x16d
  github.com/solo-io/gloo/test/kube2e/gloo_test.glob..func4.2.3()
      /home/runner/work/gloo/gloo/test/kube2e/gloo/setup_syncer_test.go:96 +0x2b0
  github.com/onsi/ginkgo/internal/leafnodes.(*runner).runSync()
      /home/runner/go/pkg/mod/github.com/onsi/[email protected]/internal/leafnodes/runner.go:113 +0xf5
  github.com/onsi/ginkgo/internal/leafnodes.(*runner).run()
      /home/runner/go/pkg/mod/github.com/onsi/[email protected]/internal/leafnodes/runner.go:64 +0x157
  github.com/onsi/ginkgo/internal/leafnodes.(*ItNode).Run()
      /home/runner/go/pkg/mod/github.com/onsi/[email protected]/internal/leafnodes/it_node.go:26 +0x94
  github.com/onsi/ginkgo/internal/spec.(*Spec).runSample()
      /home/runner/go/pkg/mod/github.com/onsi/[email protected]/internal/spec/spec.go:215 +0x359
  github.com/onsi/ginkgo/internal/spec.(*Spec).Run()
      /home/runner/go/pkg/mod/github.com/onsi/[email protected]/internal/spec/spec.go:138 +0x1ab
  github.com/onsi/ginkgo/internal/specrunner.(*SpecRunner).runSpec()
      /home/runner/go/pkg/mod/github.com/onsi/[email protected]/internal/specrunner/spec_runner.go:200 +0x15a
  github.com/onsi/ginkgo/internal/specrunner.(*SpecRunner).runSpecs()
      /home/runner/go/pkg/mod/github.com/onsi/[email protected]/internal/specrunner/spec_runner.go:170 +0x255
  github.com/onsi/ginkgo/internal/specrunner.(*SpecRunner).Run()
      /home/runner/go/pkg/mod/github.com/onsi/[email protected]/internal/specrunner/spec_runner.go:66 +0x136
  github.com/onsi/ginkgo/internal/suite.(*Suite).Run()
      /home/runner/go/pkg/mod/github.com/onsi/[email protected]/internal/suite/suite.go:79 +0x6f5
  github.com/onsi/ginkgo.runSpecsWithCustomReporters()
      /home/runner/go/pkg/mod/github.com/onsi/[email protected]/ginkgo_dsl.go:245 +0x212
  github.com/onsi/ginkgo.RunSpecsWithDefaultAndCustomReporters()
      /home/runner/go/pkg/mod/github.com/onsi/[email protected]/ginkgo_dsl.go:228 +0x257
  github.com/solo-io/gloo/test/kube2e/gloo_test.TestGloo()
      /home/runner/work/gloo/gloo/test/kube2e/gloo/gloo_suite_test.go:38 +0x41d
  testing.tRunner()
      /opt/hostedtoolcache/go/1.18.2/x64/src/testing/testing.go:1439 +0x213
  testing.(*T).Run.func1()
      /opt/hostedtoolcache/go/1.18.2/x64/src/testing/testing.go:1486 +0x47

Previous read at 0x00c0004c1c80 by goroutine 989:
  runtime.mapaccess2()
      /opt/hostedtoolcache/go/1.18.2/x64/src/runtime/map.go:456 +0x0
  k8s.io/apimachinery/pkg/runtime.(*Scheme).New()
      /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/runtime/scheme.go:318 +0xc8
  k8s.io/apimachinery/pkg/runtime.UseOrCreateObject()
      /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/runtime/codec.go:97 +0x149
  k8s.io/apimachinery/pkg/runtime/serializer/json.(*Serializer).Decode()
      /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/runtime/serializer/json/json.go:259 +0xb2a
  k8s.io/apimachinery/pkg/runtime.WithoutVersionDecoder.Decode()
      /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/runtime/helper.go:252 +0xb9
  k8s.io/apimachinery/pkg/runtime.(*WithoutVersionDecoder).Decode()
      <autogenerated>:1 +0xb5
  k8s.io/apimachinery/pkg/runtime.Decode()
      /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/runtime/codec.go:58 +0x1f4
  k8s.io/client-go/rest/watch.(*Decoder).Decode()
      /home/runner/go/pkg/mod/k8s.io/[email protected]/rest/watch/decoder.go:62 +0x18c
  k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive()
      /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/watch/streamwatcher.go:105 +0x19a
  k8s.io/apimachinery/pkg/watch.NewStreamWatcher.func1()
      /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/watch/streamwatcher.go:76 +0x39

Goroutine 40 (running) created at:
  testing.(*T).Run()
      /opt/hostedtoolcache/go/1.18.2/x64/src/testing/testing.go:1486 +0x724
  testing.runTests.func1()
      /opt/hostedtoolcache/go/1.18.2/x64/src/testing/testing.go:1839 +0x99
  testing.tRunner()
      /opt/hostedtoolcache/go/1.18.2/x64/src/testing/testing.go:1439 +0x213
  testing.runTests()
      /opt/hostedtoolcache/go/1.18.2/x64/src/testing/testing.go:1837 +0x7e4
  testing.(*M).Run()
      /opt/hostedtoolcache/go/1.18.2/x64/src/testing/testing.go:1719 +0xa71
  main.main()
      _testmain.go:47 +0x2e4

Goroutine 989 (running) created at:
  k8s.io/apimachinery/pkg/watch.NewStreamWatcher()
      /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/watch/streamwatcher.go:76 +0x216
  k8s.io/client-go/rest.(*Request).newStreamWatcher()
      /home/runner/go/pkg/mod/k8s.io/[email protected]/rest/request.go:775 +0x6e4
  k8s.io/client-go/rest.(*Request).Watch()
      /home/runner/go/pkg/mod/k8s.io/[email protected]/rest/request.go:720 +0x609
  k8s.io/client-go/kubernetes/typed/core/v1.(*configMaps).Watch()
      /home/runner/go/pkg/mod/k8s.io/[email protected]/kubernetes/typed/core/v1/configmap.go:112 +0x2b8
  k8s.io/client-go/kubernetes/typed/core/v1.ConfigMapInterface.Watch-fm()
      <autogenerated>:1 +0xcc
  github.com/solo-io/solo-kit/pkg/api/v1/clients/kube.NewSharedInformer.func2()
      /home/runner/go/pkg/mod/github.com/solo-io/[email protected]/pkg/api/v1/clients/kube/resource_client_factory.go:157 +0x3e5
  k8s.io/client-go/tools/cache.(*ListWatch).Watch()
      /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/listwatch.go:111 +0xd4
  k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch()
      /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:414 +0x7f4
  k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
      /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:221 +0x44
  k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1()
      /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155 +0x48
  k8s.io/apimachinery/pkg/util/wait.BackoffUntil()
      /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156 +0xce
  k8s.io/client-go/tools/cache.(*Reflector).Run()
      /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:220 +0x2de
  k8s.io/client-go/tools/cache.(*Reflector).Run-fm()
      <autogenerated>:1 +0x44
  k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()
      /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:56 +0x3e
  k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
      /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:73 +0x73
==================
==================
WARNING: DATA RACE
Write at 0x00c0004c1ce0 by goroutine 40:
  runtime.mapassign()
      /opt/hostedtoolcache/go/1.18.2/x64/src/runtime/map.go:578 +0x0
  k8s.io/apimachinery/pkg/runtime.(*Scheme).AddUnversionedTypes()
      /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/runtime/scheme.go:150 +0x484
  k8s.io/apimachinery/pkg/apis/meta/v1.AddToGroupVersion()
      /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/apis/meta/v1/register.go:75 +0x60a
  sigs.k8s.io/controller-runtime/pkg/scheme.(*Builder).Register.func1()
      /home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/scheme/scheme.go:73 +0xbb
  k8s.io/apimachinery/pkg/runtime.(*SchemeBuilder).AddToScheme()
      /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/runtime/scheme_builder.go:29 +0xa1
  sigs.k8s.io/controller-runtime/pkg/scheme.(*Builder).AddToScheme()
      /home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/scheme/scheme.go:87 +0x2c
  github.com/solo-io/solo-apis/pkg/api/ratelimit.solo.io/v1alpha1.AddToScheme()
      /home/runner/go/pkg/mod/github.com/solo-io/[email protected]/pkg/api/ratelimit.solo.io/v1alpha1/register.go:32 +0x5b
  github.com/solo-io/solo-apis/pkg/api/ratelimit.solo.io/v1alpha1.NewClientsetFromConfig()
      /home/runner/go/pkg/mod/github.com/solo-io/[email protected]/pkg/api/ratelimit.solo.io/v1alpha1/clients.go:52 +0x60
  github.com/solo-io/gloo/projects/gloo/pkg/api/external/solo/ratelimit.NewRateLimitClients()
      /home/runner/work/gloo/gloo/projects/gloo/pkg/api/external/solo/ratelimit/extensions.go:54 +0xf9
  github.com/solo-io/gloo/projects/gloo/pkg/syncer/setup.RunGlooWithExtensions()
      /home/runner/work/gloo/gloo/projects/gloo/pkg/syncer/setup/setup_syncer.go:504 +0xb5a
  github.com/solo-io/gloo/projects/gloo/pkg/syncer/setup.RunGloo()
      /home/runner/work/gloo/gloo/projects/gloo/pkg/syncer/setup/setup_syncer.go:417 +0x24f
  github.com/solo-io/gloo/projects/gloo/pkg/syncer/setup.(*setupSyncer).Setup()
      /home/runner/work/gloo/gloo/projects/gloo/pkg/syncer/setup/setup_syncer.go:391 +0x21f4
  github.com/solo-io/gloo/projects/gloo/pkg/syncer/setup.(*setupSyncer).Setup-fm()
      <autogenerated>:1 +0xdc
  github.com/solo-io/gloo/test/kube2e/gloo_test.glob..func4.1.1()
      /home/runner/work/gloo/gloo/test/kube2e/gloo/setup_syncer_test.go:56 +0x16d
  github.com/solo-io/gloo/test/kube2e/gloo_test.glob..func4.2.3()
      /home/runner/work/gloo/gloo/test/kube2e/gloo/setup_syncer_test.go:96 +0x2b0
  github.com/onsi/ginkgo/internal/leafnodes.(*runner).runSync()
      /home/runner/go/pkg/mod/github.com/onsi/[email protected]/internal/leafnodes/runner.go:113 +0xf5
  github.com/onsi/ginkgo/internal/leafnodes.(*runner).run()
      /home/runner/go/pkg/mod/github.com/onsi/[email protected]/internal/leafnodes/runner.go:64 +0x157
  github.com/onsi/ginkgo/internal/leafnodes.(*ItNode).Run()
      /home/runner/go/pkg/mod/github.com/onsi/[email protected]/internal/leafnodes/it_node.go:26 +0x94
  github.com/onsi/ginkgo/internal/spec.(*Spec).runSample()
      /home/runner/go/pkg/mod/github.com/onsi/[email protected]/internal/spec/spec.go:215 +0x359
  github.com/onsi/ginkgo/internal/spec.(*Spec).Run()
      /home/runner/go/pkg/mod/github.com/onsi/[email protected]/internal/spec/spec.go:138 +0x1ab
  github.com/onsi/ginkgo/internal/specrunner.(*SpecRunner).runSpec()
      /home/runner/go/pkg/mod/github.com/onsi/[email protected]/internal/specrunner/spec_runner.go:200 +0x15a
  github.com/onsi/ginkgo/internal/specrunner.(*SpecRunner).runSpecs()
      /home/runner/go/pkg/mod/github.com/onsi/[email protected]/internal/specrunner/spec_runner.go:170 +0x255
  github.com/onsi/ginkgo/internal/specrunner.(*SpecRunner).Run()
      /home/runner/go/pkg/mod/github.com/onsi/[email protected]/internal/specrunner/spec_runner.go:66 +0x136
  github.com/onsi/ginkgo/internal/suite.(*Suite).Run()
      /home/runner/go/pkg/mod/github.com/onsi/[email protected]/internal/suite/suite.go:79 +0x6f5
  github.com/onsi/ginkgo.runSpecsWithCustomReporters()
      /home/runner/go/pkg/mod/github.com/onsi/[email protected]/ginkgo_dsl.go:245 +0x212
  github.com/onsi/ginkgo.RunSpecsWithDefaultAndCustomReporters()
      /home/runner/go/pkg/mod/github.com/onsi/[email protected]/ginkgo_dsl.go:228 +0x257
  github.com/solo-io/gloo/test/kube2e/gloo_test.TestGloo()
      /home/runner/work/gloo/gloo/test/kube2e/gloo/gloo_suite_test.go:38 +0x41d
  testing.tRunner()
      /opt/hostedtoolcache/go/1.18.2/x64/src/testing/testing.go:1439 +0x213
  testing.(*T).Run.func1()
      /opt/hostedtoolcache/go/1.18.2/x64/src/testing/testing.go:1486 +0x47

Previous read at 0x00c0004c1ce0 by goroutine 783:
  runtime.mapaccess2()
      /opt/hostedtoolcache/go/1.18.2/x64/src/runtime/map.go:456 +0x0
  k8s.io/apimachinery/pkg/runtime.(*Scheme).ObjectKinds()
      /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/runtime/scheme.go:289 +0x2f6
  k8s.io/apimachinery/pkg/runtime/serializer/json.(*Serializer).Decode()
      /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/runtime/serializer/json/json.go:237 +0x54e
  k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode()
      /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/runtime/serializer/streaming/streaming.go:107 +0x569
  k8s.io/client-go/rest/watch.(*Decoder).Decode()
      /home/runner/go/pkg/mod/k8s.io/[email protected]/rest/watch/decoder.go:49 +0xbb
  k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive()
      /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/watch/streamwatcher.go:105 +0x19a
  k8s.io/apimachinery/pkg/watch.NewStreamWatcher.func1()
      /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/watch/streamwatcher.go:76 +0x39

Goroutine 40 (running) created at:
  testing.(*T).Run()
      /opt/hostedtoolcache/go/1.18.2/x64/src/testing/testing.go:1486 +0x724
  testing.runTests.func1()
      /opt/hostedtoolcache/go/1.18.2/x64/src/testing/testing.go:1839 +0x99
  testing.tRunner()
      /opt/hostedtoolcache/go/1.18.2/x64/src/testing/testing.go:1439 +0x213
  testing.runTests()
      /opt/hostedtoolcache/go/1.18.2/x64/src/testing/testing.go:1837 +0x7e4
  testing.(*M).Run()
      /opt/hostedtoolcache/go/1.18.2/x64/src/testing/testing.go:1719 +0xa71
  main.main()
      _testmain.go:47 +0x2e4

Goroutine 783 (running) created at:
  k8s.io/apimachinery/pkg/watch.NewStreamWatcher()
      /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/watch/streamwatcher.go:76 +0x216
  k8s.io/client-go/rest.(*Request).newStreamWatcher()
      /home/runner/go/pkg/mod/k8s.io/[email protected]/rest/request.go:775 +0x6e4
  k8s.io/client-go/rest.(*Request).Watch()
      /home/runner/go/pkg/mod/k8s.io/[email protected]/rest/request.go:720 +0x609
  k8s.io/client-go/kubernetes/typed/core/v1.(*configMaps).Watch()
      /home/runner/go/pkg/mod/k8s.io/[email protected]/kubernetes/typed/core/v1/configmap.go:112 +0x2b8
  k8s.io/client-go/kubernetes/typed/core/v1.ConfigMapInterface.Watch-fm()
      <autogenerated>:1 +0xcc
  github.com/solo-io/solo-kit/pkg/api/v1/clients/kube.NewSharedInformer.func2()
      /home/runner/go/pkg/mod/github.com/solo-io/[email protected]/pkg/api/v1/clients/kube/resource_client_factory.go:157 +0x3e5
  k8s.io/client-go/tools/cache.(*ListWatch).Watch()
      /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/listwatch.go:111 +0xd4
  k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch()
      /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:414 +0x7f4
  k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
      /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:221 +0x44
  k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1()
      /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155 +0x48
  k8s.io/apimachinery/pkg/util/wait.BackoffUntil()
      /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156 +0xce
  k8s.io/client-go/tools/cache.(*Reflector).Run()
      /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:220 +0x2de
  k8s.io/client-go/tools/cache.(*Reflector).Run-fm()
      <autogenerated>:1 +0x44
  k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()
      /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:56 +0x3e
  k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
      /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:73 +0x73
@kdorosh kdorosh self-assigned this Sep 21, 2022
@kdorosh
Copy link
Contributor Author

kdorosh commented Sep 21, 2022

race comes from

scheme := scheme.Scheme

@sam-heilbron
Copy link
Contributor

I previously thought I had fixed this with solo-io/gloo#6282, and then some of the HA work may have changed the tests to bring it back in.

@kdorosh
Copy link
Contributor Author

kdorosh commented Sep 22, 2022

@sam-heilbron this was actually seen in the live race image deployed in internal testing environment; not just a testing bug. It's rare but separate; the proposed fix is to add to the global scheme during init() before the k8s watcher has a chance to read using the global scheme. you can see the proposed fix as part of solo-io/gloo#7207

@kdorosh
Copy link
Contributor Author

kdorosh commented Sep 23, 2022

fix for gloo was merged. reopening until skv2 is updated with the function removed, so people don't use this possibly racy function by accident

@kdorosh kdorosh reopened this Sep 23, 2022
@kdorosh kdorosh removed their assignment Sep 27, 2022
@kdorosh
Copy link
Contributor Author

kdorosh commented Sep 29, 2022

reopening to ensure we fix and gloo mesh does not get hit by this

@kdorosh kdorosh reopened this Sep 29, 2022
@kdorosh
Copy link
Contributor Author

kdorosh commented Sep 29, 2022

we should not close this ticket until the code has been removed from skv2 so no one else introduces this by accident

@fernandosanchezs
Copy link

Seen again recently: solo-io/external-apis#34

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
4 participants