Difficulty getting feedback on score output

[zjkkhhh]

I ran the tool as follows and got no output:

java -jar renaissance-gpl-0.14.2.jar all

Later I realized that maybe I had to add --json or --csv to get the output.

java -jar renaissance-gpl-0.14.2.jar --csv results_001.csv all

Such a design may unexpectedly fail to obtain output.

Taking SPECjvm2008 released by SPEC as an example, its default output is SPECjvm2008.001.html, SPECjvm2008.001.raw, SPECjvm2008.001.txt, and some images. In this way, we can see clear results after running the benchmark once, and even have a visualized html web page.

Is it possible to consider making some improvements and optimizations to Renaissance's default output?

[ceresek]

To kick off the discussion - historical reason for this is compatibility with DaCapo harness output, which people are used to, even though it lacks a lot of the information that gets collected by the Renaissance harness. Currently, the JSON output is the most informative - perhaps we could dump that as a default, possibly with timestamp to avoid overwriting past results accidentally ?

[zjkkhhh]

To kick off the discussion - historical reason for this is compatibility with DaCapo harness output, which people are used to, even though it lacks a lot of the information that gets collected by the Renaissance harness. Currently, the JSON output is the most informative - perhaps we could dump that as a default, possibly with timestamp to avoid overwriting past results accidentally ?

I don't want to question whether the output data format is csv or json.
The main thing I want to express is: There is currently no default output format. Can Renaissance provide default output csv and json?

[zjkkhhh]

I think it would be more appropriate if the user enters the following command to output a csv file and a json file by default:

java -jar renaissance-gpl-0.14.2.jar all

[zjkkhhh]

Renaissance's parameters include <benchmark-specification>, which means "List of benchmarks (or groups) to execute (or 'all')."
There is no default value for this parameter, you must enter "all" if you want to run them all.
To expand on this question, can Renaissance omit "all"? That is, entering the following two commands is equivalent:

java -jar renaissance-gpl-0.14.2.jar

java -jar renaissance-gpl-0.14.2.jar all

[farquet]

While outputting a file by default can be convenient, we should really not run all benchmarks if none is provided. By the nature of JVMs, profile pollutions or different profiles based on the order in which benchmarks ran can greatly affect the performance of individual benchmarks. This is why JMH forks new JVMs by default for instance. Best performance insights will be gathered by running each benchmark individually.

Note that DaCapo also just shows the help message if no benchmark is specified.

Actually, I think we should provide a giant warning message when users run all benchmarks in a single VM process, similar to the JMH one when -f 0 is used.

[zjkkhhh]

While outputting a file by default can be convenient, we should really not run all benchmarks if none is provided. By the nature of JVMs, profile pollutions or different profiles based on the order in which benchmarks ran can greatly affect the performance of individual benchmarks. This is why JMH forks new JVMs by default for instance. Best performance insights will be gathered by running each benchmark individually.

Note that DaCapo also just shows the help message if no benchmark is specified.

Actually, I think we should provide a giant warning message when users run all benchmarks in a single VM process, similar to the JMH one when -f 0 is used.

Thank you very much for your reply, I understand what you mean.
Do you think outputting a file by default is a reasonable idea?

[farquet]

Do you think outputting a file by default is a reasonable idea?

Since this has no impact on benchmarking, I think whatever makes it more convenient for users is a good idea, so yes.

[vhotspur]

Do you think outputting a file by default is a reasonable idea?

The default output is to screen (standard output) which is fine for quick experimenting.

If one needs to do some serious benchmarking, they often need to fine-tune the JVM too (e.g. with -Xmx etc.) so adding another parameter for the desired output format is no big deal IMHO.

From the beginning we intentionally wanted to provide a set of workloads open for different use rather than a self-evaluating package (like SPEC does).

For example, SPEC benchmarks are always driven by their ability to provide verified results that can be published on SPEC website. We do not have ambitions to provide such a database of results but instead we expect that the users would tweak the invocation to their needs. Some people might be interested in startup costs, some in long running effects. We try not to limit them which means we do not perform any analysis and let the user choose as much options as possible.

To me, your use case seems reasonable for some initial experimenting but once you start running more heavy experiments, you would probably need to store data in well-defined filenames instead of searching the current directory for the last in the list of files (or something).

I understand that it might be surprising behavior for some people. But some people (including me) would argue that creating (persistent) files without (explicit) request is even more confusing.

Perhaps I missed some clear and frequent use case for dumping to renaissance.001.json but it seems to me this is something rarely needed and would only complicate the harness code and make the suite behavior less predictable.

[zjkkhhh]

Do you think outputting a file by default is a reasonable idea?

The default output is to screen (standard output) which is fine for quick experimenting.

If one needs to do some serious benchmarking, they often need to fine-tune the JVM too (e.g. with -Xmx etc.) so adding another parameter for the desired output format is no big deal IMHO.

From the beginning we intentionally wanted to provide a set of workloads open for different use rather than a self-evaluating package (like SPEC does).

For example, SPEC benchmarks are always driven by their ability to provide verified results that can be published on SPEC website. We do not have ambitions to provide such a database of results but instead we expect that the users would tweak the invocation to their needs. Some people might be interested in startup costs, some in long running effects. We try not to limit them which means we do not perform any analysis and let the user choose as much options as possible.

To me, your use case seems reasonable for some initial experimenting but once you start running more heavy experiments, you would probably need to store data in well-defined filenames instead of searching the current directory for the last in the list of files (or something).

I understand that it might be surprising behavior for some people. But some people (including me) would argue that creating (persistent) files without (explicit) request is even more confusing.

Perhaps I missed some clear and frequent use case for dumping to renaissance.001.json but it seems to me this is something rarely needed and would only complicate the harness code and make the suite behavior less predictable.

I'm new to the performance field, and the question may be a little naive. Thank you for your patient answers.