skaffold debug command cause panic: reflect.Value.Interface #7225

beastpu · 2022-03-24T15:21:44Z

Information

Skaffold version: v 1.37.0
Operating system: centos7
go verison: 1.18

Steps to reproduce the behavior

skaffold debug command causes panic when deploy helm charts. if i use skaffold with v1.35 version it works well. I tested other commands like skaffold deploy, and no errors were reported.

Error: INSTALLATION FAILED: error while running post render on files: error while running command /usr/local/bin/skaffold. error output:
panic: reflect.Value.Interface: cannot return value obtained from unexported field or method

goroutine 1 [running]:
reflect.valueInterface({0x1fdcd80, 0xc000bc49f0, 0xc000057770}, 0x24)
	/usr/local/Cellar/go/1.17.7/libexec/src/reflect/value.go:1375 +0xd9
reflect.Value.Interface(...)
	/usr/local/Cellar/go/1.17.7/libexec/src/reflect/value.go:1364
github.com/GoogleContainerTools/skaffold/pkg/skaffold/parser/configlocations.generateObjPointerToYAMLNodeMap({0xc0005d0240, 0x203000}, {0x1fdcd80, 0xc000bc49f0, 0x38}, {0x2287fe0, 0xc000bc49f0, 0xc000b1c290}, {0x1e86abe, 0x3}, ...)
	/tmpfs/src/github/skaffold/pkg/skaffold/parser/configlocations/configlocations.go:283 +0x825
github.com/GoogleContainerTools/skaffold/pkg/skaffold/parser/configlocations.generateObjPointerToYAMLNodeMap({0xc0005d0240, 0x203000}, {0x2287fe0, 0xc000bc49f0, 0x203000}, {0x21fb840, 0xc000bc49f0, 0xc000b1c6f0}, {0x1e86f96, 0x4}, ...)
	/tmpfs/src/github/skaffold/pkg/skaffold/parser/configlocations/configlocations.go:361 +0x105d
github.com/GoogleContainerTools/skaffold/pkg/skaffold/parser/configlocations.generateObjPointerToYAMLNodeMap({0xc0005d0240, 0x203000}, {0x22418c0, 0xc000534570, 0x38}, {0x236f1a0, 0xc000534540, 0xc000b1cb50}, {0x1f1d618, 0x5}, ...)
	/tmpfs/src/github/skaffold/pkg/skaffold/parser/configlocations/configlocations.go:361 +0x105d
github.com/GoogleContainerTools/skaffold/pkg/skaffold/parser/configlocations.generateObjPointerToYAMLNodeMap({0xc0005d0240, 0x203000}, {0x236f1a0, 0xc000534540, 0x203000}, {0x1ffeea0, 0xc00068f128, 0xc000b1cfb0}, {0xc000c020a0, 0xa}, ...)
	/tmpfs/src/github/skaffold/pkg/skaffold/parser/configlocations/configlocations.go:361 +0x105d
github.com/GoogleContainerTools/skaffold/pkg/skaffold/parser/configlocations.generateObjPointerToYAMLNodeMap({0xc0005d0240, 0x203000}, {0x1ffeea0, 0xc00068f128, 0x38}, {0x236f620, 0xc00068f100, 0xc000b1d410}, {0x1f3376a, 0x7}, ...)
	/tmpfs/src/github/skaffold/pkg/skaffold/parser/configlocations/configlocations.go:344 +0x1391
github.com/GoogleContainerTools/skaffold/pkg/skaffold/parser/configlocations.generateObjPointerToYAMLNodeMap({0xc0005d0240, 0xf4f}, {0x236f620, 0xc00068f100, 0xf4f}, {0x1ffef20, 0xc0008f28a8, 0x0}, {0xc000c02030, 0xb}, ...)
	/tmpfs/src/github/skaffold/pkg/skaffold/parser/configlocations/configlocations.go:361 +0x105d
github.com/GoogleContainerTools/skaffold/pkg/skaffold/parser/configlocations.generateObjPointerToYAMLNodeMap({0xc0005d0240, 0x0}, {0x1ffef20, 0xc0008f28a8, 0x1472c38}, {0x24067a0, 0xc0008f2700, 0xc000b1dbf8}, {0x1f3cf2a, 0x8}, ...)
	/tmpfs/src/github/skaffold/pkg/skaffold/parser/configlocations/configlocations.go:344 +0x1391
github.com/GoogleContainerTools/skaffold/pkg/skaffold/parser/configlocations.generateObjPointerToYAMLNodeMap({0xc0005d0240, 0xc000bcab18}, {0x2198a40, 0xc0008f2700, 0x0}, {0x0, 0x0, 0x2}, {0x0, 0x0}, ...)
	/tmpfs/src/github/skaffold/pkg/skaffold/parser/configlocations/configlocations.go:361 +0x105d
github.com/GoogleContainerTools/skaffold/pkg/skaffold/parser/configlocations.buildMapOfSchemaObjPointerToYAMLInfos({0xc0005d0240, 0x30}, 0xc0008f2700, 0x3f77e20, 0x0)
	/tmpfs/src/github/skaffold/pkg/skaffold/parser/configlocations/configlocations.go:208 +0x385
github.com/GoogleContainerTools/skaffold/pkg/skaffold/parser/configlocations.Parse(...)
	/tmpfs/src/github/skaffold/pkg/skaffold/parser/configlocations/configlocations.go:91
github.com/GoogleContainerTools/skaffold/pkg/skaffold/parser.GetConfigSet({_, _}, {{0x0, 0x0}, {0xc000054072, 0xd}, {0x259767f, 0x6}, {0x0, 0x0}, ...})
	/tmpfs/src/github/skaffold/pkg/skaffold/parser/config.go:101 +0x355
github.com/GoogleContainerTools/skaffold/cmd/skaffold/app/cmd.withFallbackConfig({_, _}, {_, _}, {{0x0, 0x0}, {0xc000054072, 0xd}, {0x259767f, 0x6}, ...}, ...)
	/tmpfs/src/github/skaffold/cmd/skaffold/app/cmd/runner.go:110 +0x79
github.com/GoogleContainerTools/skaffold/cmd/skaffold/app/cmd.runContext({_, _}, {_, _}, {{0x0, 0x0}, {0xc000054072, 0xd}, {0x259767f, 0x6}, ...})
	/tmpfs/src/github/skaffold/cmd/skaffold/app/cmd/runner.go:82 +0x85
github.com/GoogleContainerTools/skaffold/cmd/skaffold/app/cmd.createNewRunner({_, _}, {_, _}, {{0x0, 0x0}, {0xc000054072, 0xd}, {0x259767f, 0x6}, ...})
	/tmpfs/src/github/skaffold/cmd/skaffold/app/cmd/runner.go:62 +0x7b
github.com/GoogleContainerTools/skaffold/cmd/skaffold/app/cmd.withRunner({0x2a9b9d8, 0xc0005f0080}, {0x2a4fd60, 0xc000850be0}, 0xc000b1fa90)
	/tmpfs/src/github/skaffold/cmd/skaffold/app/cmd/runner.go:50 +0x83
github.com/GoogleContainerTools/skaffold/cmd/skaffold/app/cmd.runFilter({0x2a9b9d8, 0xc0005f0080}, {0x2a4fd60, 0xc000850be0}, 0x25, {0xc0001b0fc0, 0x2a4ef40, 0xc00000e018})
	/tmpfs/src/github/skaffold/cmd/skaffold/app/cmd/filter.go:62 +0x92
github.com/GoogleContainerTools/skaffold/cmd/skaffold/app/cmd.NewCmdFilter.func1({0x2a9b9d8, 0xc0005f0080}, {0x2a4fd60, 0xc000850be0})
	/tmpfs/src/github/skaffold/cmd/skaffold/app/cmd/filter.go:55 +0x4e
github.com/GoogleContainerTools/skaffold/cmd/skaffold/app/cmd.(*builder).NoArgs.func1(0x9b3ae0, {0xc000b1fb80, 0xc000b1fb58, 0x0})
	/tmpfs/src/github/skaffold/cmd/skaffold/app/cmd/commands.go:138 +0x66
github.com/GoogleContainerTools/skaffold/cmd/skaffold/app/cmd.applyPostRunHooks.func1(0xc0005298c0, {0xc000850b90, 0x0, 0x5})
	/tmpfs/src/github/skaffold/cmd/skaffold/app/cmd/commands.go:185 +0x2f
github.com/spf13/cobra.(*Command).execute(0xc0005298c0, {0xc000850b40, 0x5, 0x5})
	/Users/kbuilder/go/pkg/mod/github.com/spf13/[email protected]/command.go:856 +0x60e
github.com/spf13/cobra.(*Command).ExecuteC(0xc00084e000)
	/Users/kbuilder/go/pkg/mod/github.com/spf13/[email protected]/command.go:974 +0x3bc
github.com/spf13/cobra.(*Command).ExecuteContextC(...)
	/Users/kbuilder/go/pkg/mod/github.com/spf13/[email protected]/command.go:911
github.com/GoogleContainerTools/skaffold/cmd/skaffold/app.Run({0x2a4ef40, 0xc00000e018}, {0x2a4ef40, 0xc00000e020})
	/tmpfs/src/github/skaffold/cmd/skaffold/app/skaffold.go:50 +0x2ad
main.main()
	/tmpfs/src/github/skaffold/cmd/skaffold/skaffold.go:49 +0x1e5
: exit status 2
Cleaning up...

The text was updated successfully, but these errors were encountered:

tejal29 · 2022-03-24T18:42:11Z

@beastpu Can you please point us to your skaffold file or a small reproduction case?

tejal29 · 2022-03-24T18:44:57Z

/cc @aaron-prindle I see a bunch of TODO regarding errors Vs panic. Maybe this issue is exercising that code.

beastpu · 2022-03-25T10:27:08Z

@beastpu Can you please point us to your skaffold file or a small reproduction case?

skaffold.yaml

apiVersion: skaffold/v2beta28
kind: Config
metadata:
  name: dev
build:
  artifacts:
  - image: gate
    custom:
      buildCommand: make build-gate-image-debug
  tagPolicy:
    sha256: {}
  local:
    push: false
    useBuildkit: true
    concurrency: 0
deploy:
  helm:
    releases:
    - name: test
      chartPath: deploy/helm
      artifactOverrides:
        gate.image: gate
      setValueTemplates:
        filePath: '{{ .filePath }}'
      imageStrategy:
        helm: {}
    flags:
      upgrade:
      - --install
profiles:
- name: debug
  patches:
  - op: replace
    path: /build/artifacts
    value:
    - custom:
        buildCommand: make build-gate-image-debug
      image: gate

skaffold debug -p debug

I commented the profile code

apiVersion: skaffold/v2beta28
kind: Config
metadata:
  name: dev
build:
  artifacts:
    - image: gate
      custom:
        buildCommand: make build-gate-image-debug
  tagPolicy:
    sha256: {}
  local:
    push: false
    useBuildkit: true
    concurrency: 0
deploy:
  helm:
    releases:
    - name: test
      chartPath: deploy/helm
      artifactOverrides:
        gate.image: gate
      setValueTemplates:
        filePath: '{{ .filePath }}'
      imageStrategy:
        helm: {}
    flags:
      upgrade:
        - --install
#profiles:
#- name: debug
#  patches:
#  - op: replace
#    path: /build/artifacts
#    value:
#    - custom:
#        buildCommand: make build-gate-image-debug
#      image: gate

run skaffold debug it works well

make build-gate-image-debug just build docker image. the panic error information will be reported when i run skaffold debug -p debug or skaffold debug command, if i comment profile code it works well.
I guess it has something to do with specifying a profile or not @tejal29 @aaron-prindle

wollowizard · 2022-04-03T17:39:01Z

Got the same error. also for me, after commenting out the profiles section, it works fine.
Got it after I updated to v1.37.1, from v1.35.2

aaron-prindle · 2022-04-07T01:17:18Z

I was able to reproduce this building off of skaffold/example/templated-fields and adding a profile and then running:
skaff debug --default-repo=gcr.io/my-repo

The only change I made was to modify the skaffold.yaml there to look like the following:

apiVersion: skaffold/v2beta25
kind: Config
metadata:
  name: my-app
build:
  artifacts:
  - image: skaffold-templated
    custom:
      buildCommand: "./build.sh"
      dependencies:
        paths:
        - "**/*.go"
        - go.mod
deploy:
  helm:
    releases:
    - name: skaffold-templated
      chartPath: charts
      artifactOverrides:
        image: skaffold-templated
      setValueTemplates:
        imageRepo: "{{.IMAGE_REPO}}"
        imageTag: "{{.IMAGE_TAG}}"
profiles:
- name: debug
  patches:
  - op: replace
    path: /build/artifacts
    value:
    - image: skaffold-templated-debug
      custom:
        buildCommand: "./build-debug.sh"
      dependencies:
        paths:
        - "**/*.go"
        - go.mod

When I attempted the simple fix of removing the logic that is currently panicking (the if statement with v.Interface()and the visited map are not critical, they jus check for recursive loops in the config but that is not very likely anyway) it seems that a different piece of code - transform.go then panics:
https://github.com/GoogleContainerTools/skaffold/blob/main/pkg/skaffold/kubernetes/debugging/transform.go#L52

full logs for transform.go panic when running skaff debug --default-repo=gcr.io/my-repo:
https://gist.github.com/aaron-prindle/4ee7e1d31976a855b46ba844c06626f6

panic log snippet

$ skaffold debug --default-repo=gcr.io/my-repo

Helm release skaffold-templated not installed. Installing...
Error: error while running post render on files: error while running command /usr/local/bin/skaff. error output:
fault address 0xffffffffffffffff is to an unmapped Go arena - you may be accessing
an object or string in an arena after that arena was freed

unexpected fault address 0xffffffffffffffff
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0xffffffffffffffff pc=0xf98042]

goroutine 1 [running]:
runtime.throw({0x365927e?, 0xc000d2bc20?})
	/usr/lib/google-golang/src/runtime/panic.go:998 +0x71 fp=0xc0005de020 sp=0xc0005ddff0 pc=0x5ae791
runtime.sigpanic()
	/usr/lib/google-golang/src/runtime/signal_unix.go:831 +0x1e5 fp=0xc0005de050 sp=0xc0005de020 pc=0x5c5725
github.com/modern-go/reflect2.(*UnsafeMapIterator).HasNext(0xc000e1b218)
	/usr/local/google/home/aprindle/go/pkg/mod/github.com/modern-go/[email protected]/unsafe_map.go:125 +0x22 fp=0xc0005de068 sp=0xc0005de050 pc=0xf98042
github.com/json-iterator/go.(*sortKeysMapEncoder).IsEmpty(0xc000d92f60, 0xc000b769b0)
	/usr/local/google/home/aprindle/go/pkg/mod/github.com/json-iterator/[email protected]/reflect_map.go:334 +0x51 fp=0xc0005de0a8 sp=0xc0005de068 pc=0xff0a11
github.com/json-iterator/go.(*placeholderEncoder).IsEmpty(0xc000d90e20, 0xc000b769b0)
	/usr/local/google/home/aprindle/go/pkg/mod/github.com/json-iterator/[email protected]/reflect.go:336 +0x3e fp=0xc0005de0d0 sp=0xc0005de0a8 pc=0xfe71de
github.com/json-iterator/go.(*structFieldEncoder).IsEmpty(0xc000d9ee70, 0xc000b76920)
	/usr/local/google/home/aprindle/go/pkg/mod/github.com/json-iterator/[email protected]/reflect_struct_encoder.go:118 +0x62 fp=0xc0005de100 sp=0xc0005de0d0 pc=0x1003662
github.com/json-iterator/go.(*structEncoder).Encode(0xc000d9efc0, 0xc000b76920, 0xc0005ce600)
	/usr/local/google/home/aprindle/go/pkg/mod/github.com/json-iterator/[email protected]/reflect_struct_encoder.go:148 +0x30a fp=0xc0005de218 sp=0xc0005de100 pc=0x1003aaa
github.com/json-iterator/go.(*structFieldEncoder).Encode(0xc000e8c6f0, 0xc000b76900, 0xc0005ce600)
	/usr/local/google/home/aprindle/go/pkg/mod/github.com/json-iterator/[email protected]/reflect_struct_encoder.go:110 +0x7f fp=0xc0005de2f0 sp=0xc0005de218 pc=0x10033df
github.com/json-iterator/go.(*structEncoder).Encode(0xc000e8c7e0, 0xc000b76900, 0xc0005ce600)
	/usr/local/google/home/aprindle/go/pkg/mod/github.com/json-iterator/[email protected]/reflect_struct_encoder.go:158 +0x3a5 fp=0xc0005de408 sp=0xc0005de2f0 pc=0x1003b45
github.com/json-iterator/go.(*OptionalEncoder).Encode(0xc000e890b0, 0xc000696ec0, 0xc0005ce600)
	/usr/local/google/home/aprindle/go/pkg/mod/github.com/json-iterator/[email protected]/reflect_optional.go:70 +0x52 fp=0xc0005de438 sp=0xc0005de408 pc=0xff89d2
github.com/json-iterator/go.(*onePtrEncoder).Encode(0xc000e890c0, 0xc000b76900, 0xc0005ce600)
	/usr/local/google/home/aprindle/go/pkg/mod/github.com/json-iterator/[email protected]/reflect.go:219 +0x87 fp=0xc0005de468 sp=0xc0005de438 pc=0xfe62a7
github.com/json-iterator/go.(*Stream).WriteVal(0xc0005ce600, {0x361c8c0, 0xc000b76900})
	/usr/local/google/home/aprindle/go/pkg/mod/github.com/json-iterator/[email protected]/reflect.go:98 +0xed fp=0xc0005de4d0 sp=0xc0005de468 pc=0xfe4f8d
github.com/json-iterator/go.(*frozenConfig).Marshal(0xc0003dc640, {0x361c8c0, 0xc000b76900})
	/usr/local/google/home/aprindle/go/pkg/mod/github.com/json-iterator/[email protected]/config.go:299 +0x125 fp=0xc0005de638 sp=0xc0005de4d0 pc=0xfd7745
k8s.io/apimachinery/pkg/runtime/serializer/json.(*Serializer).doEncode(0xc000118780, {0x3a355a8, 0xc000b76900}, {0x3a25160, 0xc00087cb00})
	/usr/local/google/home/aprindle/go/pkg/mod/k8s.io/[email protected]/pkg/runtime/serializer/json/json.go:306 +0xe9 fp=0xc0005de770 sp=0xc0005de638 pc=0x145f609
k8s.io/apimachinery/pkg/runtime/serializer/json.(*Serializer).Encode(0xc000118780, {0x3a355a8, 0xc000b76900}, {0x3a25160, 0xc00087cb00})
	/usr/local/google/home/aprindle/go/pkg/mod/k8s.io/[email protected]/pkg/runtime/serializer/json/json.go:301 +0x1ae fp=0xc0005de810 sp=0xc0005de770 pc=0x145f4ae
github.com/GoogleContainerTools/skaffold/pkg/skaffold/kubernetes/debugging.glob..func1({0x3a355a8, 0xc000b76900})
	/usr/local/google/home/aprindle/skaffold/pkg/skaffold/kubernetes/debugging/transform.go:52 +0x105 fp=0xc0005de8c8 sp=0xc0005de810 pc=0x2c77085

This panic is less obvious to me what might be happening, I don't think it is related to configlocations.go code. Need more time to understand what might have changed here from v1.35.2 -> v1.37.0 and what the root cause is

briandealwis · 2022-04-07T03:17:29Z

@aaron-prindle If I cherry-pick your change in #7276 on HEAD, I'm able to launch skaffold debug. I tried bisecting the change and ended up on @halvards's #6563 which introduced debug support for Ko images. So I think this is somehow tickling a bug somewhere in the k8s.io modules, which was fixed with @halvards's updates to k8s.io v0.21.6 in #7224.

aaron-prindle · 2022-04-07T17:31:16Z

Thanks @briandealwis for the investigation. I have confirmed I also see this is fixed using my changes from HEAD. We will merge #7276 once tests pass and this should be resolved

tejal29 assigned aaron-prindle Mar 24, 2022

tejal29 added kind/bug Something isn't working triage/needs-information Information requested labels Mar 24, 2022

aaron-prindle mentioned this issue Apr 7, 2022

fix: add panic recovery logic to reflection logic for yaml line number info #7276

Merged

aaron-prindle closed this as completed in #7276 Apr 7, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

skaffold debug command cause panic: reflect.Value.Interface #7225

skaffold debug command cause panic: reflect.Value.Interface #7225

beastpu commented Mar 24, 2022

tejal29 commented Mar 24, 2022

tejal29 commented Mar 24, 2022

beastpu commented Mar 25, 2022 •

edited

Loading

wollowizard commented Apr 3, 2022

aaron-prindle commented Apr 7, 2022 •

edited

Loading

briandealwis commented Apr 7, 2022

aaron-prindle commented Apr 7, 2022

skaffold debug command cause panic: reflect.Value.Interface #7225

skaffold debug command cause panic: reflect.Value.Interface #7225

Comments

beastpu commented Mar 24, 2022

Information

Steps to reproduce the behavior

tejal29 commented Mar 24, 2022

tejal29 commented Mar 24, 2022

beastpu commented Mar 25, 2022 • edited Loading

wollowizard commented Apr 3, 2022

aaron-prindle commented Apr 7, 2022 • edited Loading

briandealwis commented Apr 7, 2022

aaron-prindle commented Apr 7, 2022

beastpu commented Mar 25, 2022 •

edited

Loading

aaron-prindle commented Apr 7, 2022 •

edited

Loading