cmd/pprof: sync with upstream https://github.com/google/pprof ?

[dgryski]

The pprof tool was initially an import of Google's internal version, which is now on GitHub. Can we just switch to that new repo? Are there patches in Go's copy that need to be pushed upstream to make google/pprof work with current Go binaries?

Maybe this ends up just being a doc fix saying "you can also use this other pprof tool over here".

[hyangah]

I ask @rauls5382 most pprof questions

[rauls5382]

I am not aware of any changes to pprof that are missing on the github version.

There are some UI differences that I plan to smooth out, but other than that it should be fully functional with Go programs. Please open issues at github.com/google/pprof/issues if you encounter any problems.

I think it would be great if we could sync up the two versions at some point.

[bradfitz]

I am not aware of any changes to pprof that are missing on the github version.

Have you been merging all Go's changes? There have been a handful.

[rauls5382]

I have merged a few in the past but admittedly haven't checked recently. I will take a new look.

[quentinmit]

@rauls5382 Have you had a chance to compare-and-merge pprof?

[rauls5382]

I finally spent some time on this. There's about a dozen small CLs that should be upstreamed.
Most of them are very small. I'll be upstreaming them shortly.

[rsc]

@rauls5382, it sounds like you found changes in Go that need to be merged into google/pprof.

What about the other direction? Are there patches in google/pprof that need to be merged into Go?

[rauls5382]

Yes, google/pprof has had many modifications after the golang fork.
There was a significant reorganization of the packages before they were published into golang/pprof. The process of syncing up with upstream will require some manual changes as the package structure is different. I'll put a proposal together.

[quentinmit]

Too late for Go 1.8. Please try to have this ready for Go1.9. Our pprof is falling behind and I would love to have the improvements in Go.

[gopherbot]

CL https://golang.org/cl/33212 mentions this issue.

[hyangah]

@rauls5382 We are now in early stage of go1.9 cycle. Can we merge into it?

I checked the abandoned cl/33212 and it seems that it doesn't include some patches added in Go side. So need refresh.

[hyangah]

For long term maintenance, wouldn't it be easier to use vendoring if the canonical source remains in github.com/google/pprof, and ask developers to update the upstream directly (then update the vendorred version in go repo)?

[rauls5382]

I have just sent a new commit for review: https://go-review.googlesource.com/#/c/36552/ Please take a look. I've tried to include all Go changes still relevant. But if you see anything missing please bring it up on the review. The patches in the Go side that you found missing are recent ones? I do think that moving forward we should ask developers to update upstream directly, and make sure that the version in the Go distro is refreshed from upstream regularly (at a minimum once per Go release). Sadly it currently requires some manual work (eg to edit the package paths and remove unnecessary dependences such as C++ demangling). *Raúl E. Silvera* Software Engineer waymo.com <http://www.google.com/url?q=http%3A%2F%2Fwww.waymo.com&sa=D&sntz=1&usg=AFQjCNGA45CYjWapdB9ifiVJ8bxRUJu2mw>

…

On Tue, Feb 7, 2017 at 3:45 PM, Hyang-Ah Hana Kim ***@***.***> wrote: For long term maintenance, wouldn't it be easier to use vendoring if the canonical source remains in github.com/google/pprof, and ask developers to update the upstream directly (then update the vendorred version in go repo)? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#16184 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AKqb8oMU2WfWngOu9kDfBK2XsR8BgCIOks5raQIegaJpZM4I-e91> .

[dgryski]

Would it be possible to use Go's build tags features to toggle C++ demangling, binutils usage, etc so that the google/pprof repo can be more canonically upstream with no changes when it's imported to the Go repo?

[davecheney]

I like this idea.

…

On Wed, Feb 8, 2017 at 8:34 PM, Damian Gryski ***@***.***> wrote: Would it be possible to use Go's build tags features to toggle C++ demangling, binutils usage, etc so that the google/pprof repo can be more canonically upstream with no changes when it's imported to the Go repo? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#16184 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAcA4aOswKA-bj7EH-VeERzOLX_yzbgks5raYwtgaJpZM4I-e91> .

[hyangah]

I tried to compare the package and source code structure of Go's pprof and github.com/google/pprof.

Go --> github.com/google/pprof
src/internal/pprof/* --> profile/*
cmd/pprof/internal/* --> internal/* except binutil

Hopefully just import path change would be sufficient.

cmd/pprof/*.go files (package main) contain custom driver.Fetcher, driver.ObjTool implementations and go tool compatible documentation. So these need to be committed directly to the go repo(?) or as @dgryski proposed, let google/pprof repo host them in a special directory.

[rsc]

The cmd/pprof directory should probably just be left in the main repo. It's only the helper libraries (subdirectories and src/internal/pprof) that should be pulled down from github.com/google/pprof.

[rauls5382]

Right, so it would be: cmd/pprof in the Go repo cmd/pprof/internal/* copy from github.com/google/pprof/internal/* internal/pprof/profile copy from github.com/google/pprof/profile I can move the binutils package in the github.com/google/pprof repo to a different directory to simplify. The upstream pprof has a dependence into github.com/ianlancetaylor/demangle to demangle C++ names. Is there value on adding that dependence to the golang distro, or should we continue skipping it? *Raúl E. Silvera* Software Engineer waymo.com <http://www.google.com/url?q=http%3A%2F%2Fwww.waymo.com&sa=D&sntz=1&usg=AFQjCNGA45CYjWapdB9ifiVJ8bxRUJu2mw>

…

On Wed, Feb 8, 2017 at 10:57 AM, Russ Cox ***@***.***> wrote: The cmd/pprof directory should probably just be left in the main repo. It's only the helper libraries that should be pulled down from github.com/google/pprof. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#16184 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AKqb8jSoDU7Yf7VTHNwr8gXX6IzJk77Kks5rahAOgaJpZM4I-e91> .

[rsc]

For cmd/pprof, I think the way forward is to copy the relevant directories into go/src/cmd/vendor/github.com/google/pprof/xxx, with the files unmodified. Update cmd/vendor/vendor.json when you do. Then cmd/pprof becomes just a single directory with a few files, that imports paths like "github.com/google/pprof/xxx", which are resolved into the cmd/vendor directory. It is fine to vendor iant's demangle too.

For the purposes of updating cmd/pprof, let's forget that go/src/internal/pprof/profile exists. Vendor github.com/google/pprof/profile into go/src/cmd/vendor/github.com/google/pprof/profile like all the other directories, and let pprof use it.

We'll maintain go/src/internal/pprof/profile as a separate fork; we only use it for writing profiles, so we can cut out even more and just live with the minimal maintenance burden. This code is linked into essentially every Go program, so we want to be able to make it as small as possible and not have it be subject to unexpected dependencies.

Thanks.

[bradfitz]

I believe this happened (a047f72, etc). Closing.

@rauls5382, correct me if I'm wrong.