Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python stack unwinding fails, but with null error #2314

Closed
Garbett1 opened this issue Nov 29, 2023 · 3 comments
Closed

Python stack unwinding fails, but with null error #2314

Garbett1 opened this issue Nov 29, 2023 · 3 comments

Comments

@Garbett1
Copy link

Describe the bug
Parca agent stack unwinding fails to add the unwind table, but also has a null error which is surprising.

To Reproduce
Can't replicate with a minimal example yet, but happens consistently with the specific binary.

Logs

kubectl logs -n parca parca-agent-rb55j | grep 18846

level=debug name=parca-agent ts=2023-11-29T12:45:32.972851349Z caller=cpu.go:536 component=cpu_profiler msg="adding unwind tables" pid=18846
level=error name=parca-agent ts=2023-11-29T12:45:32.975991544Z caller=cpu.go:557 component=cpu_profiler msg="failed to add unwind table" pid=18846 err=null
level=debug name=parca-agent ts=2023-11-29T12:45:33.170072189Z caller=cpu.go:531 component=cpu_profiler msg="prefetching process info" pid=18846
level=debug name=parca-agent ts=2023-11-29T12:45:33.170377246Z caller=cpu.go:536 component=cpu_profiler msg="adding unwind tables" pid=18846
level=error name=parca-agent ts=2023-11-29T12:45:33.173663715Z caller=cpu.go:557 component=cpu_profiler msg="failed to add unwind table" pid=18846 err=null
level=debug name=parca-agent ts=2023-11-29T12:45:35.182409458Z caller=cpu.go:531 component=cpu_profiler msg="prefetching process info" pid=18846
level=debug name=parca-agent ts=2023-11-29T12:45:35.182471246Z caller=cpu.go:536 component=cpu_profiler msg="adding unwind tables" pid=18846
level=error name=parca-agent ts=2023-11-29T12:45:35.184726234Z caller=cpu.go:557 component=cpu_profiler msg="failed to add unwind table" pid=18846 err=null
level=debug name=parca-agent ts=2023-11-29T12:45:44.121114761Z caller=cpu.go:531 component=cpu_profiler msg="prefetching process info" pid=18846
level=debug name=parca-agent ts=2023-11-29T12:45:44.12125953Z caller=cpu.go:536 component=cpu_profiler msg="adding unwind tables" pid=18846
level=error name=parca-agent ts=2023-11-29T12:45:44.123850309Z caller=cpu.go:557 component=cpu_profiler msg="failed to add unwind table" pid=18846 err=null
level=debug name=parca-agent ts=2023-11-29T12:46:20.493137114Z caller=cpu.go:531 component=cpu_profiler msg="prefetching process info" pid=18846
level=debug name=parca-agent ts=2023-11-29T12:46:20.530767211Z caller=info.go:304 component=process_info msg="interpreter information fetched" interpreter=Python version=3.10.13 pid=18846
level=debug name=parca-agent ts=2023-11-29T12:46:20.53203637Z caller=maps.go:851 component=cpu_profiler component=bpf_maps msg="Python Version Offset" pid=18846 version_offset_index=1
level=debug name=parca-agent ts=2023-11-29T12:46:20.532145376Z caller=cpu.go:536 component=cpu_profiler msg="adding unwind tables" pid=18846
level=error name=parca-agent ts=2023-11-29T12:46:20.534201669Z caller=cpu.go:557 component=cpu_profiler msg="failed to add unwind table" pid=18846 err=null
level=debug name=parca-agent ts=2023-11-29T12:46:37.889658547Z caller=cpu.go:531 component=cpu_profiler msg="prefetching process info" pid=18846
level=debug name=parca-agent ts=2023-11-29T12:46:37.889877709Z caller=cpu.go:536 component=cpu_profiler msg="adding unwind tables" pid=18846
level=error name=parca-agent ts=2023-11-29T12:46:37.892896043Z caller=cpu.go:557 component=cpu_profiler msg="failed to add unwind table" pid=18846 err=null

Software (please complete the following information):

  • Parca Agent Version: ghcr.io/parca-dev/parca-agent:main-1701258441-d828aaf4

** Workload:**

  • Runtime (if applicable): Python 3.10.13
    ** Environment :**

  • Linux Distribution: COS

  • Linux Version: 5.15.120+

  • Arch: amd64

  • Kubernetes Version (if applicable): Server Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.13-gke.200", GitCommit:"f24d4fe14dcec505c2b37cc1e5b7024d971f6360", GitTreeState:"clean", BuildDate:"2023-08-25T09:26:18Z", GoVersion:"go1.20.7 X:boringcrypto", Compiler:"gc", Platform:"linux/amd64"}

  • Container Runtime (if applicable): containerd 1.6.21

@brancz
Copy link
Member

brancz commented Nov 29, 2023

Huh, this is kind of odd.

level.Debug(p.logger).Log("msg", "adding unwind tables", "pid", pid)
if err := p.bpfMaps.AddUnwindTableForProcess(pid, nil, true); err == nil {
// Happy path.
return
}
// Error handling,
switch {
case errors.Is(err, bpfmaps.ErrNeedMoreProfilingRounds):
p.metrics.unwindTableAddErrors.WithLabelValues(labelNeedMoreProfilingRounds).Inc()
level.Debug(p.logger).Log("msg", "PersistUnwindTable called to soon", "err", err)
case errors.Is(err, os.ErrNotExist):
p.metrics.unwindTableAddErrors.WithLabelValues(labelProcfsRace).Inc()
level.Debug(p.logger).Log("msg", "failed to add unwind table due to a procfs race", "pid", pid, "err", err)
case errors.Is(err, bpfmaps.ErrTooManyExecutableMappings):
p.metrics.unwindTableAddErrors.WithLabelValues(labelTooManyMappings).Inc()
level.Warn(p.logger).Log("msg", "failed to add unwind table due to having too many executable mappings", "pid", pid, "err", err)
case errors.Is(err, buildid.ErrTextSectionNotFound):
p.processErrorTracker.Track(pid, err)
default:
p.metrics.unwindTableAddErrors.WithLabelValues(labelOther).Inc()
level.Error(p.logger).Log("msg", "failed to add unwind table", "pid", pid, "err", err)
}

if err == nil we should already be returning before 🤔

@Tatskaari
Copy link

Are you falling a fowl of the typed nil variable nonsense?

@brancz
Copy link
Member

brancz commented Nov 29, 2023

Yes, definitely possible!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants