New Epic: Exhaust scripts

[jmcook1186]

Hi IFers,

I'm outlining here an initial draft spec for a some upcoming work so you have an opportunity to comment and help us refine the spec before the work begins.

Background

We currently handle exporting data to yaml, csv or console using a combination of commands and manifest config. We support:

• export to yaml (by passing --output <savepath> on the cli and adding outputs: - yaml to the manifest),
• export to CSV (by passing --output <savepath> with a trailing hashtag and the parameter to export on the cli, plus output: - csv in the manifest
• export to the console (by passing --stdout and no config in the manifest).

Other export formats have to be handled using plugins.

Getting the combinations of config and command correct can be quite error-prone. It's not a great user experience, and it also adds some friction to our automated testing process (as there are in-manifest differences to handle).

During the IF refactor and the hackathon we experimented with other exhaust formats, such as moving everything to dedicated exhaust plugins. However, this creates additional complexity because it introduces requirements for certain plugins to be configured and executed at specific moments during the IF execution, which feels very fragile. With plugins we can't configure the exhaust easily on the command line. Exhaust plugins also muddy the waters when it comes to the manifest file being used as an executable audit of environmental impacts. The exhaust is irrelevant to the actual numbers - it is presentation only - it just adds noise to the manifest file. Finally, with exhaust being handled by plugins, every time you want to change how your data is presented you have to execute and recalculate your whole manifest file, which is inefficient.

Description of proposed changes

The proposed solution to these issues is to handle exhaust using scripts instead of plugins.

This cements IF as purely an engine for computing and communicating environmental impacts, and presentation related tasks are moved out of IF and into dedicated scripts. IF can only generate output files - any subsequent operations over that output data is handled externally to IF using dedicated scripts. This way, the manifest file is set up to become a protocol, and the outputs can be piped to any arbitrary downstream post-processing program.

We can first assert that IF outputs yaml data. This can be to a file or to the console. No additional code is required to configure this. Either the IF receives --stdout to trigger dumping the yaml data to the console or it receives --output to trigger saving to a yaml file. If both commands are passed, IF does both.

For exporting to other formats, let's take the example of csv-export.

Instead of being a plugin that operates during the IF execution, csv-export can become a separate script that operates on the IF output yaml data. We can refer to this script as if-csv.

if-csv can be invoked on the command line, taking an IF output file as an argument. Instead of having to create a fiddly command using a hashtag to define the parameter to export as we do today, the script can just expose its own commands to do the same thing more intuitively, for example:

if-csv -in-file <example.yml> --fields carbon --output output.csv`

The above example will export the carbon parameter from the output data in example.yml and save it to output.csv.

This can be used to run the exhaust script over an ie output without having to re-execute ie.

However, in situations where we do want to re-execute ie, we cna simply pipe ie into the if-csv script in a single command:

ie -m <manifest> --stdout | if-csv --fields carbon --output output.csv

We intend for if-csv to be bundled into the IF downlaod, but you can develop your own exhaust scripts with any arbitrary logic and use it in the same way. We'll provide some boilerplate code and a tutorial to help.

Specific tasks

The specific tasks that we'll undertake to compelte this change will be written up as detailed tickets on our board, but for now in this preliminary stage, here's what I expect us to do:

• Make —stdout the default export option if no output option is configured via the CLI

• Update export to yaml an IF option where all the config is provided on the CLI not in the manifest. Providing a savepath should be all that’s required to trigger saving to yaml ie -m <manifest> -o <savepath> . If both --stdout and --output are provided then save to file and print to console.

• Add some logic to check file extension provided in savepath on the CLI and gracefully handle cases where the extension is provided, is not provided and is provided incorrectly.

• Refactor csv-export and csv-export-raw features from plugins to scripts, if-csv

• Refine arguments to if-csv so that it no longer relies on having extensionless filenames and parameters identified with a hashtag and instead uses clear commands such as

  if-csv --fields **energy** --file output.yaml

• Update docs with description of exhaust scripts shipped with IF and tutorial on how to write an exhaust script

How you can help

You can help by reading through this post and raising any feedback, comments or concerns as comments here. Later on, I'll also link out to individual tickets as we create them and you can let us know if you want to work on one - there may be some core changes we want to protect for the core team only, but in general we are keen to open up development on the IF to the community. Keep your eyes peeled!

@jawache @zanete @narekhovhannisyan @MariamKhalatova @manushak

[narekhovhannisyan]

@jmcook1186 if-csv script is included in if framework and package.json file?

[jmcook1186]

yes i think it should be.