-
Notifications
You must be signed in to change notification settings - Fork 613
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Option to select source files directory #262
Comments
You can use pprof -source_path=/home/marko/lakafka ... to specify the path for your source tree. |
Actually, it is listed in pprof -help, so no need for any changes. |
I think -source_path currently does not help to locate source files that have their path recorded as absolute path in the debug information which seems to be the case here. See pprof/internal/report/source.go Line 572 in e82ee9a
I also find the logic of handling the search paths here a bit inconvenient: it attempts to append the file path from the debug information to each non-empty path prefix of each search path. So if one specifies
@rauls5382 I am curious what are the use cases for this search strategy? Generally, looking up the content in the parent directories of the search paths the user specified is not a very good thing as some of those parent directories could be network automount directories and looking up random paths in those is super-slow. What I think I would prefer as a user instead, is trying every possible suffix of the path being searched, breadth-first. So for my example, the following paths would be tried:
That avoids the issue with the lookup of arbitrary paths in parent directories, Also, with that, I'd also remove the check for whether the path is absolute. This would make the reported case supported as the user could specify now
|
Alternatively, we could make the behavior of the
which I think is inconvenient which is why GDB also has the option to configure substitution rules, see the same link. The solution I suggested may be a simpler compromise. |
The idea behind the current search is to allow the user to specify a path inside their copy of the source tree and have things work, without having to finely line up mount points. This works pretty well with the default ($CWD), so if you're standing anywhere inside a copy of the source tree things just work. I agree that we should fix the handling of absolute paths. Perhaps adding an option for some simple substitution? About trimming profile paths from the left, I would worry about there being multiple files in the tree with the same set of trailing directories. Say you have a tree with: util/config.h I would want to avoid resolving cool-library/util/config.h to util/config.h |
Allowing to specify an arbitrary path within the source root of interest doesn't sound too convincing of a heuristic to me unless it enables some use case rather than having just a utility function. And in some cases it could result in unusual behavior - e.g. if there is a source file with relative path like If there is a tree |
I also have the problem of running pprof on a machine that is not the build machine, and having absolute paths in the profile. Here is an example stack frame from a profile I am studying: 0xe5543e k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage.(*cacheWatcher).process+0x36e /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/cacher.go:947So far, I have been able to hack the machine running pprof so that the build paths exist on my pprof machine --- but I do not enjoy doing this. Would it make sense to introduce an option to simply ignore the absolute paths and do a lookup based on the source_path and that other identifier? |
This is an issue for me too. Our Go binaries are produced on a CI server, and I often take profiles from production but there's no way afaik to get the Source view. Maintainers (@rauls5382 ?) Is there an acceptable functional design, such that I could contribute a Pull Request? |
Apologies about the radio silence. Yeah, time to figure this one out. My preference is still what I suggested in #262 (comment) since it addresses every use case I can think of with just a slight tweak of the current behavior of -search_path and does not introduce new flags. @rauls5382 Objections to that? |
The use case that I want to make sure is preserved is when the user's cwd is somewhere inside their local build tree and the profile has relative paths. Today that 'just works' without having to cd to the root of the local tree or specifying the source_path option. I think this proposal will break that flow. I think the fundamental problem is that we do not know how much of the remote path is irrelevant an should be stripped. Could we perhaps implement some heuristics and provide an option to override? Say that if the user specifies source_tree=/my/tree/lakafka and we see remote paths that include /.../lakafka/... we could automatically strip the remote paths to the left. |
I see -- I agree breaking the current behavior is not a good idea. The proposed heuristic might be not reliable if the anchor directory name is something ambiguous like "src". Another solution is introduce a new option like you suggested before. Something like This could also have some limited support for globs to allow something like |
SGTM. I agree that a heuristic can be defeated but we would have an option override in the (hopefully infrequent) corner cases. How about if instead of a full rewrite_path option we have a --remote_source_path that says what we should strip from absolute paths to turn them into relative paths, and then let source_path work as it does today. Have remote_source_path default to a good guess based on the source_path. My concern with rewrite_path is how it interacts with source_path. I'd rather have something more orthogonal. |
Despite the radio silence, I do have a change for this, will send it soon. |
@aalexand If you need any help, or testing or code review give me a bell. I am very motivated to get this change. We are running a fork of pprof right now to be able to see source built on other machines, and I would love to be able to use the mainline version. |
When pprof is asked to show annotated source for a profile collected on another machine for a Go program, the profile contains absolute paths which may not exist on the local machine. In that case pprof currently fails to locate the source with no option to help it. The new option adds a way to specify one or several source path prefixes that should be trimmed from the source paths in the profile before applying the search using search path option. For example, taking the example from the issue where the source file path in the profile is /home/teamcitycpp/agent09/work/56cbaf90675b10ff/_gopath/src/badoo/lakafka/main.go and the local path is /home/marko/lakafka/main.go. The user would specify `-trim-path=/home/teamcitycpp/agent09/work/56cbaf90675b10ff/_gopath/src/badoo` to make pprof find the source. The source path doesn't need to be specified if the current working dir is anything at or under `/home/marko/`. TODO: Heuristical default value for the trim path. Fixes google#262.
When pprof is asked to show annotated source for a profile collected on another machine for a Go program, the profile contains absolute paths which may not exist on the local machine. In that case pprof currently fails to locate the source with no option to help it. The new option adds a way to specify one or several source path prefixes that should be trimmed from the source paths in the profile before applying the search using search path option. For example, taking the example from the issue where the source file path in the profile is /home/teamcitycpp/agent09/work/56cbaf90675b10ff/_gopath/src/badoo/lakafka/main.go and the local path is /home/marko/lakafka/main.go. The user may specify `-trim-path=/home/teamcitycpp/agent09/work/56cbaf90675b10ff/_gopath/src/badoo` to make pprof find the source. The source path doesn't need to be specified if the current working dir is anything at or under `/home/marko/`. When the trim path is not specified, it is guessed heuristically based on the basename of configured searched paths. In the example above, setting `-search-path=/home/marko/lakafka` would be sufficient to activate the heuristic successfully. Or having the current directory as `/home/marko/lakafka` since the search path is by default set to the current working directory. Note that the heuristic currently does not attempt to walk the configured search paths up like the search does. This is to keep it simple, use `-trim-path` explicitly in more complicated cases. Fixes google#262.
When pprof is asked to show annotated source for a profile collected on another machine for a Go program, the profile contains absolute paths which may not exist on the local machine. In that case pprof currently fails to locate the source with no option to help it. The new option adds a way to specify one or several source path prefixes that should be trimmed from the source paths in the profile before applying the search using search path option. For example, taking the example from the issue where the source file path in the profile is /home/teamcitycpp/agent09/work/56cbaf90675b10ff/_gopath/src/badoo/lakafka/main.go and the local path is /home/marko/lakafka/main.go. The user may specify `-trim-path=/home/teamcitycpp/agent09/work/56cbaf90675b10ff/_gopath/src/badoo` to make pprof find the source. The source path doesn't need to be specified if the current working dir is anything at or under `/home/marko/`. When the trim path is not specified, it is guessed heuristically based on the basename of configured searched paths. In the example above, setting `-search-path=/home/marko/lakafka` would be sufficient to activate the heuristic successfully. Or having the current directory as `/home/marko/lakafka` since the search path is by default set to the current working directory. Note that the heuristic currently does not attempt to walk the configured search paths up like the search does. This is to keep it simple, use `-trim-path` explicitly in more complicated cases. Fixes google#262.
When pprof is asked to show annotated source for a profile collected on another machine for a Go program, the profile contains absolute paths which may not exist on the local machine. In that case pprof currently fails to locate the source with no option to help it. The new option adds a way to specify one or several source path prefixes that should be trimmed from the source paths in the profile before applying the search using search path option. For example, taking the example from the issue where the source file path in the profile is /home/teamcitycpp/agent09/work/56cbaf90675b10ff/_gopath/src/badoo/lakafka/main.go and the local path is /home/marko/lakafka/main.go. The user may specify `-trim-path=/home/teamcitycpp/agent09/work/56cbaf90675b10ff/_gopath/src/badoo` to make pprof find the source. The source path doesn't need to be specified if the current working dir is anything at or under `/home/marko/`. When the trim path is not specified, it is guessed heuristically based on the basename of configured searched paths. In the example above, setting `-search-path=/home/marko/lakafka` would be sufficient to activate the heuristic successfully. Or having the current directory as `/home/marko/lakafka` since the search path is by default set to the current working directory. Note that the heuristic currently does not attempt to walk the configured search paths up like the search does. This is to keep it simple, use `-trim-path` explicitly in more complicated cases. Fixes google#262.
Add "trim path" option which can be used to relocate sources. When pprof is asked to show annotated source for a profile collected on another machine for a Go program, the profile contains absolute paths which may not exist on the local machine. In that case pprof currently fails to locate the source with no option to help it. The new option adds a way to specify one or several source path prefixes that should be trimmed from the source paths in the profile before applying the search using search path option. For example, taking the example from the issue where the source file path in the profile is /home/teamcitycpp/agent09/work/56cbaf90675b10ff/_gopath/src/badoo/lakafka/main.go and the local path is /home/marko/lakafka/main.go. The user may specify `-trim-path=/home/teamcitycpp/agent09/work/56cbaf90675b10ff/_gopath/src/badoo` to make pprof find the source. The source path doesn't need to be specified if the current working dir is anything at or under `/home/marko/`. When the trim path is not specified, it is guessed heuristically based on the basename of configured searched paths. In the example above, setting `-search-path=/home/marko/lakafka` would be sufficient to activate the heuristic successfully. Or having the current directory as `/home/marko/lakafka` since the search path is by default set to the current working directory. Note that the heuristic currently does not attempt to walk the configured search paths up like the search does. This is to keep it simple, use `-trim-path` explicitly in more complicated cases. Fixes #262.
@aalexand, why I got this error? I run:
But when I run
Why |
@ilya-korotya Please file a separate report with instructions on how to reproduce. |
…#366) Add "trim path" option which can be used to relocate sources. When pprof is asked to show annotated source for a profile collected on another machine for a Go program, the profile contains absolute paths which may not exist on the local machine. In that case pprof currently fails to locate the source with no option to help it. The new option adds a way to specify one or several source path prefixes that should be trimmed from the source paths in the profile before applying the search using search path option. For example, taking the example from the issue where the source file path in the profile is /home/teamcitycpp/agent09/work/56cbaf90675b10ff/_gopath/src/badoo/lakafka/main.go and the local path is /home/marko/lakafka/main.go. The user may specify `-trim-path=/home/teamcitycpp/agent09/work/56cbaf90675b10ff/_gopath/src/badoo` to make pprof find the source. The source path doesn't need to be specified if the current working dir is anything at or under `/home/marko/`. When the trim path is not specified, it is guessed heuristically based on the basename of configured searched paths. In the example above, setting `-search-path=/home/marko/lakafka` would be sufficient to activate the heuristic successfully. Or having the current directory as `/home/marko/lakafka` since the search path is by default set to the current working directory. Note that the heuristic currently does not attempt to walk the configured search paths up like the search does. This is to keep it simple, use `-trim-path` explicitly in more complicated cases. Fixes google#262.
What version of pprof are you using?
tip from https://github.com/google/pprof
What operating system and processor architecture are you using?
Linux 64bit
I would like to have a way to tell pprof where it can find source files.
Imagine I have some service running on the server and I have a binary that is running there.
I can use pprof like this:
Where /tmp/lakafka-1.22.0-278 is a binary built on a build server and /tmp/prof is a profile collected via wget from http://.../debug/pprof/profile.
If I then go to View -> Source I don't see any source code. pprof is looking for the source code in the path from build server, e.g. /home/teamcitycpp/agent09/work/56cbaf90675b10ff/_gopath/src/badoo/lakafka/main.go
But I am not on a build server. I have the same source code at, lets say, /home/marko/lakafka.
I would like to tell pprof to look for the source code at /home/marko/lakafka.
Similar to what dir command does in GDB.
Thank you!
The text was updated successfully, but these errors were encountered: