Fix a bug where meta-indicators are dropped during region-processing

[danielhuppmann]

This PR fixes a bug where meta-indicators are dropped in a specific combination of arguments. The culprit is fixed specifically in 71d7ec7 (and no, it wasn't a pyam bug).

The subsequent commit refactors the apply-code to only operate on the timeseries data and add meta-indicators at the end of the loop.

closes #227

[danielhuppmann]

fyi @phackstock

[phackstock]

Ah right, thanks for the quick PR, seems like there is an issue indeed.

[phackstock]

Coming to think of it, if the processing only uses meta preserving operations, is this a pyam bug?

[phackstock]

Thanks for the PR, looks good, just three small comments.
One clarifying question from my side is how did we lose the meta indicators in the first place?
Was it the concatenation step or the aggregation? And is it something that needs to be addressed on the pyam side as well or is the behavior as expected and it was simply not correct here on the nomenclature side of things?

[danielhuppmann]

The problem of the dropped meta-indicators was caused by the line

common_region_df.append(aggregate_df._data[index])

in the compare-and-merge between aggregated and original (provided) data.

If there was no common-region-data, an empty IamDataFrame (i.e., no meta indicators) was merged with only the timeseries data (_data) of the aggregated IamDataFrame.

This was fixed with the commit 71d7ec7, merging both IamDataFrame instances, ensuring that meta indicators were carried over.

However, this bug made me realize that the previous implementation did a lot of concatenating of IamDataFrame instances with identical meta indicators. Hence the refactoring where only the timeseries data is added to _processed_data in each region-processing step, and then concatenated and cast to a new IamDataFrame instance with the meta indicators added explicitly using

processed_dfs.append(IamDataFrame(_data, meta=model_df.meta))

[phackstock]

If there was no common-region-data, an empty IamDataFrame (i.e., no meta indicators) was merged with only the timeseries data (_data) of the aggregated IamDataFrame.

Ah of course, we did hit an edge case there with the GENIE upload where we only had aggregated data.

[phackstock]

Good to be merged (apart from the sticker)