external addon: script to combine events

[GitMensch]

This is possibly a quite "special" request: I'd like to combine events that match a prefix/suffix, which would be specified on command line.

By "combine" I mean that the events would be calculated as if they would have been seen in the caller:
Giving the following four stacktraces during the pgcollect run:

main
  func1
    func1_
       intfunc 

main
  func1
     intfunc2

main
  func2
     intfunc

main
  func2
     func2_

And a suggested call of pgconvert -d symbol -c "intfunc*,*_" filename.pgdata > callgrind.out.overfiew_pgdata
"-c func count as caller, may use an asterisk to match multiple ones"

The "inspected" call stacks would be (after combining everything that starts with "intfunc" or ends with "_"):

main
  func1

main
  func1

main
  func2

main
  func2

when combining the call all costs attached to this specific entry would be counted as happened in the caller... _ maybe_ that means the combination switch must be handled (and therefore specified) during recording already?

[ostash]

Won't KCacheGrind allow you to see it the way you want? It shows you the total number of samples in func1 and func2 including samples in callees.

[GitMensch]

The issue in KCachegrind is the source view where many code lines have a reference to calling a lot of functions.
If the combined calls are all calculated to the place where they are called them those lines only have a single reference.

If not guessed already: the code is generated and include multiple functions in each line.

[GitMensch]

Having a look at some Kcachegrind visualizations I propose an additional feature: --combine-below. The reason is that there are often some external library calls where you know "I can't do anything about that but - if at all - reduce the calls".
It would be nice to be able to combine these traces as it both provides a better overview and allows to distinguish calls that are done by that function to calls that are done from own code. Layout that may shows this better:

main
  db_call
     db_func1
        db_func2
           strcmp

main
  func_1 
     db_call
        db_func10
           db_func18
              db_func22
                  memcpy

main
  stcmp

main
  func_1
     memcpy

with an assumed call of --combine-below=db_call:

main
  db_call

main
  func_1 
     db_call

main
  stcmp

main
  func_1
     memcpy

The result would be that all cost are noted as "self" in db_call (instead of "inclusive") and when inspecting the callers of strcmp one would only see the code that can be adjusted (in that sample "main").
This would even be useful for -d object because then those calls would be attributed to the program, not to the libc object.

I think the biggest point of this issue is to provide the logic and calculation to combine calls, the other points are more about the "way it is used". I think --combine-below would be the easier one to use and is likely useful for many scenarios.

[ostash]

Regarding --combine-below: what if the code under "below" calls back to the main code?

TBH, I'm not sure whether such things should be feature of pgconvert or KCacheGrind. This should be up to GUI how to represent the data.

[ostash]

Generally speaking, I don't see this as a feature of pgconvert. Callgrind format is easy, text, and documented: https://valgrind.org/docs/manual/cl-format.html. I'm ok with accepting some external Python (or any other language) script, which can rearrange things in the callgrind file.

pgconvert is already quite a complicated tool, so let it do its single job: convert collected samples to callgrind file without any alterations.

[GitMensch]

I had not inspected the format in detail and that will likely take a while. Let's keep that issue open.
To get some "spec" on the addons already: I suggest to have a sub-folder "addons" where any useful scripts may be added. I suggest that for those to be accepted they should have an interpreter line, be marked as git-executable and must include a copyright and license notice which can be different from the repo's license.

What is your take on this?

[ostash]

Makes perfect sense: addons or scripts folder, executable permission, shebang line. Different license is OK

[GitMensch]

Do you know of scripts that adjust the callgrind profiles? I did not found any, only find parsers and only found one that seems useful for the callgrind files generated by pgconvert: https://github.com/jrfonseca/gprof2dot/blob/944151e0b4466caa349fb969c1525d9b0f7b9496/gprof2dot.py#L1666-L1981 - but this is reading only, so both re-adjusting the objects and writing them out again are missing...

[ostash]

I never bothered to look :) I always wanted to have a full and unmodified view of profile data.