Debugging

Using debugger

It is possible to launch a debugger while harvester is running. First you need to define on which port the debugger listens in the master section in panda_harvester.cfg.

# port number of debugger
debugger_port = 19550

You can enter the debugger by sending a signal to a running harvester process.

$ kill -USR1 `cat $PWD/tmp.pid`

where you need to use your pid file to get the correct process ID. Back to the shell, you can then connect to the debugger using the following code snippet.

from trepan import client
client.start_client({'HOST': '127.0.0.1', 'open': True, 'PORT':19550, 'IO': 'TCP'})

where you need to give the port number of the debugger which is defined in panda_harvester.cfg. For example

$ python
>>> from trepan import client
>>> client.start_client({'HOST': '127.0.0.1', 'open': True, 'PORT':19550, 'IO': 'TCP'})
Connected.
(lib/python2.7/site-packages/pandaharvester/harvesterbody/master.py:356 @175): catch_debug
R=> None
(trepan2*)

Now you can interactively see threads, frames, backtraces, and so on, in the trepan prompt. All debug commands are explained in the trepan's page. For example,

(trepan2*) info threads
----------------------------------------
=> <_MainThread(MainThread, started 139652981257984)>
    run(self=<trepan.processor.command.info_subcmd.threads.InfoThread instance at 0x7f0375b64368>, args=[])...)
     called from file '/mnt/tmaeno/harvester/lib/python2.7/site-packages/trepan/processor/command/info_subcmd/threads.py' at line 176
----------------------------------------
   <CredManager(Thread-1, started 139652832794368)>
    wait(self=<Condition(<thread.lock object at 0x7f037e501870>, 29)>, timeout=1)
     called from file '/mnt/tmaeno/usr/lib/python2.7/threading.py' at line 359
...

(trepan2*) frame Thread-20
(/data/tmaeno/usr/lib/python2.7/threading.py:359 @344): wait
<- 359                     _sleep(delay)
R=> None 

(trepan2*) bt
->
0 wait(self=<Condition(<thread.lock object at 0x7f037e501870>, 29)>, timeout=1)
     called from file '/mnt/tmaeno/usr/lib/python2.7/threading.py' at line 359
##
1 wait(self=<threading._Event object at 0x7f0378689c90>, timeout=1)
     called from file '/mnt/tmaeno/usr/lib/python2.7/threading.py' at line 614
##
2 sleep(interval=60, stop_event=<threading._Event object at 0x7f0378689c90>, randomize=False)
     called from file '/mnt/tmaeno/harvester/lib/python2.7/site-packages/pandaharvester/harvestercore/core_utils.py' at line 96
...

Use Ctrl + C, to exit from the debugger. Once the debugger gets started, signal handlers are replaced and thus you have to kill -9 to stop the harvester process.

If gdb and python debugging extensions are available on your system, you can do the same using gdb as explained in this page.

Profiling

Harvester support statistic, deterministic, or thread-aware profiling. Statistic profiling is done with python's standard profilers, while deterministic or thread-aware profiling is done with the pprofile package which needs to be installed using pip:

$ pip install pprofile

Harvester is launched with a profiler if the --profile_output option is given to master.py. The option specifies the filename where the results of the profiler are dumped. If profiling is in the deterministic or thread-aware mode and the filename starts with "cachegrind.out", the results are written in the callgrind profile format which allows the file to be browsed with kcachegrind. If profiling is in the statistic mode, the dumo file can be analyzed using python's standard pstats package. Profiling is in the statistic mode by default, and can be changed with the --profile_mode option, "d" for the deterministic mode and "t" for the thread-aware mode. You can find detailed explanations about profiling modes in the pprofile's page. Note that the dump file is produced only when harvester is properly terminated with the USR2 or TERM signal, i.e.,

$ kill -USR2 `cat $PWD/tmp.pid`

or

$ kill `cat $PWD/tmp.pid`  

---