bpo-44895: Introduce PYTHONDUMPREFSFILE variable for refcount dumping

[corona10]

https://bugs.python.org/issue44895

[corona10]

@vstinner @pablogsal
if you have better ideas about variable naming, please let me know.
Another option might be providing option as PYTHONDUMPREFS=2.

[pablogsal]

Hummm, I would prefer not to have more environment variables. In this case is very easy to just dump the contents to a file via shell redirection ( 2> my_file). Maybe @vstinner thinks otherwise :)

[corona10]

shell redirection ( 2> my_file)

Hmm, I feel inconvenient while investigating this issue lol (It can be relative to each person. It can be only me feeling ;))
I got the idea from the JVM heap dump extracting feature and I thought that it can be convenient to people who want to extract refcount information likewise JVM people doing.

Out of curiosity, how do you extract the dump when you have to execute the python multiple times with repeat command.
For example

 repeat 10 ./python.exe -m test -R 3:3 test_exceptions -m test_no_hang_on_context_chain_cycle2 -m test_recursion_normalizing_infinite_exception -m test_recursion_in_except_handler -m test_recursion_normalizing_with_no_memory

Sorry I am not shell expert :(

[corona10]

And if we feel the adding variable is a burden how about separating PYTHONDUMPREFS option?

PYTHONDUMPREFS=1 -> stderr
PYTHONDUMPREFS=2 -> local file dump

But if we feel a burden about codebases, it's another discussion :)

[vstinner]

I suggest to use the pid rather than the timestamp. An alternative is that the environment variable contains the filename, so the user handles filename conflict. The problem is that sometimes, the correct directory is removed once the process completes. It is the case when a regrtest worker process completes.

PYTHONDUMPFILE=/path/to/dump.txt

[corona10]

Pid looks better, I would like to use PYTHONDUMPDIR instead of PYTHONDUMPFILE.
So the file will be created as PYTHONDUMPDIR/cpython-tracerefs-<:pid>.txt

[vstinner]

I don't see how dump_refs is set to non-zero.

I would prefer to call _Py_PrintReferences(fp); fclose(fp); here. So both options are not exclusive.

[corona10]

Because we have to call _Py_PrintReferenceAddresses(fp) also.
And python_dump_dir is optional.

[pablogsal]

Out of curiosity, how do you extract the dump when you have to execute the python multiple times with repeat command.

I use a shell for loop:

for i in seq 10; do ./python.exe -m test -R 3:3 test_exceptions -m test_no_hang_on_context_chain_cycle2 2>./dump_$i; done

[pablogsal]

And if we feel the adding variable is a burden how about separating PYTHONDUMPREFS option?

How do you specify the name of the file then?

I am not fully opposed to this, but I see the dumping to file option much more constrained, specially given that this is a very niche option that is not used a lot. An example is precisely what you pointed out. What is an easy way to dump to different files in a repeat or loop run? You need to modify the environment variable every time, which feels a bit odd to me.

On the other hand, if others find this useful, then is fine to me.

[corona10]

How do you specify the name of the file then?

My first implementation was creating a single file based on timestamp, which now is based on PID and I intended to create a file on where the interpreter run. This was what I intended when I suggested separating PYTHONDUMPREFS.
Since Victor suggests providing a directory option, now the file will be created on the designated directory.

IMHO, I don't prefer to designate a specific file name because the user feels inconvenient to set the name.
So I prefer generating names automatically which can distinguish the purpose.

[vstinner]

What is an easy way to dump to different files in a repeat or loop run? You need to modify the environment variable every time, which feels a bit odd to me.

I don't see what's wrong with setting an env var to invoke a command. You can reuse shell variable, like the loop counter. You cannot do that if the filename is generated.

for i in seq 10; do PYTHONDUMPREFS=loop-$i.txt ./python.exe -m test -R 3:3 test_exceptions -m test_no_hang_on_context_chain_cycle2; done

If the filename is generated, how do you use it in your loop? Example:

for i in seq 10; do
  output=/tmp/loop-$i.txt
  PYTHONDUMPREFS=$output ./python.exe -m test -R 3:3 test_exceptions -m test_no_hang_on_context_chain_cycle2
  grep "leak" $output && break
done

[corona10]

I don't see what's wrong with setting an env var to invoke a command. You can reuse shell variable, like the loop counter. You cannot do that if the filename is generated.

Hmm, I try to reduce bash level manipulation.

If the filename is generated, how do you use it in your loop? Example:

Yeah, in that case, we can not grep the file name. but we may be able to print filename through stderr instead in that case we can grep it :)

Now dumps the file .... $(PYTHONDUMPDIR)/cpython-tracerefs-<pid>.txt

[corona10]

My intention is like this

export PYTHONDUMPREFS=2 # if we do not use PYTHONDUMPDIR variable.
for i in seq 10; do
  ./python.exe -m test -R 3:3 test_exceptions -m test_no_hang_on_context_chain_cycle2
done

[corona10]

bash

#!/bin/bash
export PYTHONDUMPREFS=1 # if we do not use PYTHONDUMPDIR variable.
export PYTHONDUMPDIR="./log"
for i in 1 2 3 4 5 6 7 8 9 10
  do
    ./python.exe -m test -R 3:3 test_exceptions -m test_no_hang_on_context_chain_cycle2
  done

output

0:00:00 load avg: 2.56 Run tests sequentially
0:00:00 load avg: 2.56 [1/1] test_exceptions
beginning 6 repetitions
123456
......

== Tests result: SUCCESS ==

1 test OK.

Total duration: 419 ms
Tests result: SUCCESS
Now dumps the file: ./log/python-tracerefs-37503.txt
0:00:00 load avg: 2.56 Run tests sequentially
0:00:00 load avg: 2.56 [1/1] test_exceptions
beginning 6 repetitions
123456
......

== Tests result: SUCCESS ==

1 test OK.

Total duration: 342 ms
Tests result: SUCCESS
Now dumps the file: ./log/python-tracerefs-37523.txt
0:00:00 load avg: 2.56 Run tests sequentially
0:00:00 load avg: 2.56 [1/1] test_exceptions
beginning 6 repetitions
123456
......

== Tests result: SUCCESS ==

1 test OK.

Total duration: 342 ms
Tests result: SUCCESS
Now dumps the file: ./log/python-tracerefs-37529.txt
0:00:00 load avg: 2.56 Run tests sequentially
0:00:00 load avg: 2.56 [1/1] test_exceptions
beginning 6 repetitions
123456
......

== Tests result: SUCCESS ==

1 test OK.

Total duration: 345 ms
Tests result: SUCCESS
Now dumps the file: ./log/python-tracerefs-37530.txt
0:00:00 load avg: 2.56 Run tests sequentially
0:00:00 load avg: 2.56 [1/1] test_exceptions
beginning 6 repetitions
123456
......

== Tests result: SUCCESS ==

1 test OK.

Total duration: 365 ms
Tests result: SUCCESS
Now dumps the file: ./log/python-tracerefs-37531.txt
0:00:00 load avg: 2.56 Run tests sequentially
0:00:00 load avg: 2.56 [1/1] test_exceptions
beginning 6 repetitions
123456
......

== Tests result: SUCCESS ==

1 test OK.

Total duration: 345 ms
Tests result: SUCCESS
Now dumps the file: ./log/python-tracerefs-37533.txt
0:00:00 load avg: 2.51 Run tests sequentially
0:00:00 load avg: 2.51 [1/1] test_exceptions
beginning 6 repetitions
123456
......

== Tests result: SUCCESS ==

1 test OK.

Total duration: 381 ms
Tests result: SUCCESS
Now dumps the file: ./log/python-tracerefs-37534.txt
0:00:00 load avg: 2.51 Run tests sequentially
0:00:00 load avg: 2.51 [1/1] test_exceptions
beginning 6 repetitions
123456
......

== Tests result: SUCCESS ==

1 test OK.

Total duration: 403 ms
Tests result: SUCCESS
Now dumps the file: ./log/python-tracerefs-37535.txt
0:00:00 load avg: 2.51 Run tests sequentially
0:00:00 load avg: 2.51 [1/1] test_exceptions
beginning 6 repetitions
123456
......

== Tests result: SUCCESS ==

1 test OK.

Total duration: 371 ms
Tests result: SUCCESS
Now dumps the file: ./log/python-tracerefs-37545.txt
0:00:00 load avg: 2.51 Run tests sequentially
0:00:00 load avg: 2.51 [1/1] test_exceptions
beginning 6 repetitions
123456
......

== Tests result: SUCCESS ==

1 test OK.

Total duration: 367 ms
Tests result: SUCCESS
Now dumps the file: ./log/python-tracerefs-37546.txt

[vstinner]

I don't know any program which gives the choice where a generated filename is created. Either the program requires a filename, or it generates a filename.

Example: valgrind --log-file=<filename> expects a filename. By default, output is written into stdout (or stderr, I don't recall).

[vstinner]

Example: strace -o FILENAME.

Hum, I don't recall which programs generate filenames for the output. It is not common on Unix. Maybe there is a reason? ;-)

[corona10]

It is not common on Unix.

Hmm okay, let's change it to use designated file name :)

[vstinner]

LGTM. I just left a minor suggestion for the doc.