proposal: net/http/pprof: support finer-grained delta computation

[nsrip-dd]

The /debug/pprof endpoint supports a "seconds" parameter. When a non-CPU profile is requested with that parameter, the endpoint returns a profile representing the difference ("delta") between two of those profiles taken "seconds" apart. The current delta implementation computes that difference for every value in a sample. In particular, the heap profile will contain differences for alloc_space and alloc_objects, as well as the inuse_space and inuse_objects counts.

On the one hand, I can see how seeing the change in inuse_space or inuse_objects could aid memory leak debugging. On the other hand, there's another use case for getting the delta profiles: aggregation. The Datadog and Google Cloud Profiler client libraries compute deltas, but only for the alloc_space and alloc_objects values in a heap profile. For the inuse values, the most recent value is used rather than the difference. Those values are preferable for computing averages across several profiles.

Both of those libraries use runtime/pprof and manipulate the profiles locally to get the desired result. But for profiling services by pulling from the /debug/pprof endpoints, you'd need to do something like pull two profiles and compute the delta later. Perhaps there can be an option to request delta profiles where some of the sample values have deltas computed, but others don't? Admittedly, this would primarily be an optimization to avoid having to request multiple profiles. See this example for how it might be implemented, by adding another query parameter delta_value to specifies which values in the profile to compute deltas for.

[seankhliao]

just to be clear, what would actually change (or be added) to support this

cc @cherrymui @rsc

[nsrip-dd]

@seankhliao I've updated the issue to clarify my suggested change, adding a query parameter to control which values in a sample have deltas computed when the "seconds" parameter is also provided.

[ianlancetaylor]

CC @neild @bradfitz

[gkampitakis]

I got also confused by this detail when looking at inuse_space and was noticing negative values on my profile, not realising this was a delta profile.