Penalty for secondary energy share deviation in sectors

[Renato-Rodrigues]

Purpose of this PR

• Add support for penalization methods to incentivize the model to maintain similar secondary energy shares across sectors.
• Addresses issue #322

Features:

• Implemented support for three penalization methods that work independently of sector-specific formulations regarding fuel use:
  • Square share penalty
  • Square deviation from average share penalty
  • Min-max deviation from average share penalty

Detailed documentation can be found within the code:

*** cm_seFeSectorShareDevMethod "Switch to enable an optimization incentive for sectors to have similar shares of secondary energy fuels and determine the method used for the incentive." 
*** Possible values: off or the method name (sqSectorShare, sqSectorAvrgShare, or minMaxAvrgShare)
***  off               "The model can freely allocate bio/syn/fossil fuels between sectors. If not off, a penalization term is added so sectors are incentivized to apply similar shares of bio-fuels, synfuels, and fossils in each sector."
***  sqSectorShare     "Square share penalty"
***  sqSectorAvrgShare "Square deviation from average share penalty"
***  minMaxAvrgShare   "Min-max deviation from average share penalty"
*** The relative effect of the penalization term in the objective function is scaled to avoid affecting optimization results. This scaling factor can be defined using the switch c_seFeSectorShareDevScale.
$setglobal cm_seFeSectorShareDevMethod  sqSectorAvrgShare !! def = sqSectorAvrgShare  !! regexp = off|sqSectorShare|sqSectorAvrgShare|minMaxAvrgShare
*** c_seFeSectorShareDevUnit "Defines if the penalization term is applied over fuel shares or energy units." 
***  share,  "The square penalization is applied directly to the share values. This results in different-sized regions having varying relative penalization incentives, but the range of penalization values will be more consistent from the solver's perspective."
***  energy, "The square penalization is applied to the share values multiplied by the energy demand. This approach scales penalizations better across different-sized regions, but there is a higher risk of the penalizations being ignored and the shares not being enforced if the value range is too small."
$setglobal c_seFeSectorShareDevUnit  share !! def = share  !! regexp = share|energy

parameter
  c_seFeSectorShareDevScale "scale factor in the objective function of the penalization to incentive sectors to have similar shares of secondary energy fuels."
;
  c_seFeSectorShareDevScale = 1e-3;  !! def = 1e-3
*'

• The default option is set to sqSectorAvrgShare, as it provides the most stable solution so far with acceptable solving time.

• The sqSectorShare and sqSectorAvrgShare deviations can be defined using either share units or energy units by setting the option with the switch c_seFeSectorShareDevUnit. The default is set to share.

• The penalization term in the objective function is scaled to avoid affecting optimization results. This scaling factor can be defined using the switch c_seFeSectorShareDevScale.

Additional Changes:

• Replaced vm_demFeSector with vm_demFeSector_afterTax in the core share equations, as the marginal effect of the constraints on these shares should not be reflected in final energy marginal prices reported by REMIND.
• Disabled cm_INCONV_PENALTY_FESwitch to avoid competitive behavior between both penalization methods

Type of change

• New feature
• Bug fix
• Refactoring
• Fundamental change

Checklist:

• My code follows the coding etiquette
• I performed a self-review of my own code
• I explained my changes within the PR, particularly in hard-to-understand areas
• All automated model tests pass (FAIL 0 in the output of make test)

Further information (optional):

• Test runs are here:
  /p/projects/ecemf/REMIND/2040_scenarios/v09_2024_08_02_REMIND_3.3.1/output
/p/projects/ecemf/REMIND/2040_scenarios/v11_2024_09_02

[JakobBD]

Looks good to me. This may be a problem in process-based chemicals. But it seems to be important to the overall model, and I now know that it's there, so we can either live with it, or find a solution for this implementation and chemicals-feedstocks-related SE differences to live alongside each other. Maybe I'll approach you for that in the coming months.

On an unrelated note:
From the documentation, I'd guess that
$\text{sqSectorShare}=\sum_{i=1}^n s_i^2\quad$ with the shares $s_i$ and $\quad\sum_{i=1}^n s_i=1,\quad$ and
$\text{sqSectorAvrgShare}=\sum_{i=1}^n (s_i-\bar{s_i})^2=\sum_{i=1}^n (s_i-\frac{1}{n})^2=\sum_{i=1}^n (s_i^2-\frac{2}{n}s_i+\frac{1}{n^2})=\text{sqSectorShare}-\frac{2}{n}+\frac{1}{n}$,
so they'd only differ by a constant, which shouldn't make a difference to the solver.

Is that so, or did I get the documentation wrong, or did I mess up in my calculations?

[JakobBD]

@robertpietzcker nevermind, I messed up:
The averaging obviously isn't done over the SE types $i$, but over the sectors $j$. So $\sum_{i=1}^n s_{i,j}=1$, but $\sum_{j=1}^n s_{i,j}\neq1$.

But that might raise a different. more severe issue (if I'm not messing up again):

The penalty sqSectorShare not only penalizes unequal distribution over sectors, but also unequal overall shares of SE types.
I.e. a scenario with 100% fossil in all sectors would get a higher penalty than a scenario with 1/3 fos/bio/syn in all sectors.

So in the fossil-dominated world we live in, syn and bio would get a push with no founding.

Is that an effect we want from that penalty switch?

[robertpietzcker]

The penalty sqSectorShare not only penalizes unequal distribution over sectors, but also unequal overall shares of SE types.

yes, which is one of the reasons I prefer the (currently default) "sqSectorAvrgShare " :-)

but you are right, some info about this should be added in the description of "sqSectorShare" - like "this method ALSO creates an unwanted incentive for the model to have equal fos/syn/bio shares"

[JakobBD]

which is one of the reasons I prefer the (currently default) "sqSectorAvrgShare " :-)

To me, that seems like the only difference to sqSectorAvrgShare.

I.e.: sqSectorShare can be shown to be sqSectorAverageShare plus a penalty on unequal overall SE shares.

You're saying this only difference is unwanted (and I agree).

Which begs the questions: Is there any advantage to it? If not, then I'd be in favour of deleting it.

(As always, if I'm not messing up).

Calculations:

$\bar{s_i} = \frac{1}{n_j} \sum_j s_{ij}$

and then

$\sum_{i,j} (s_{i,j}-\bar{s_i})^2$

$= \sum_{i,j} (s_{i,j}^2- 2 s_{i,j} \bar{s_i} + \bar{s_i}^2) $

$= \sum_{i,j} s_{i,j}^2 - 2 \sum_{i} \bar{s_i} \sum_{j} s_{i,j} + \sum_{i,j} \bar{s_i}^2$

$= \sum_{i,j} s_{i,j}^2 - 2 n_j \sum_{i} \bar{s_i}^2 + n_j \sum_{i} \bar{s_i}^2 $

$= \sum_{i,j} s_{i,j}^2 - n_j \sum_{j} \bar{s_i}$

[Renato-Rodrigues]

some info about this should be added in the description of "sqSectorShare"

39c2641

[Renato-Rodrigues]

Which begs the questions: Is there any advantage to it? If not, then I'd be in favour of deleting it.

The alternative formulations are present to compare different levels of complexity and allow runtime comparisons.

The default option selected considers the pros and cons observed so far about these alternatives, but as these formulations should be further improved in subsequent PRs, I prefer to keep the current options in the code.

[Renato-Rodrigues]

@robertpietzcker
I disabled the shares incentives in this PR so we can merge this code quickly without causing any changes to REMIND.
This will allow me to do quicker tests with the latest REMIND versions in a more effective way.

Please let me know if you are ok with merging in this way so I can go ahead with the merge.

[robertpietzcker]

to make sure I understand correctly - the merged version would have as default that the model sees NONE of the penalizing terms, but just the share calculation terms?
If yes, then I would say please go ahead with the merge - but just as a safeguard, please also set up a quick before/after comparison cascade in SSP2 so we are sure that the share-calculation doesn't have an unwanted consequence that we didn't think of :-)

[Renato-Rodrigues]

please also set up a quick before/after comparison cascade in SSP2 so we are sure that the share-calculation doesn't have an unwanted consequence that we didn't think of :-)

Here is a comparison scenario of the NPi runs (1) without any code from the PR and (2) with the code from the PR but with cm_seFeSectorShareDevMethod disabled (scenarios with the PR suffix in their names) .

/p/projects/ecemf/REMIND/2040_scenarios/_mergeTest/remind_seFeSectorSharePenalization/compScen-NPI-2024-10-22_15.48.18-*.pdf

There is juts one small difference that is active between the runs. I initialized the .l share values for p_shSeFe, v_shSeFe, p_shSeFeSector and v_shSeFeSector to use the total share of bio/syn/fossil carriers to see if the initial point alone would make much difference to the results.

The results are almost the same, but there are some differences on the biomass trade and the initial point change also affects bunker emissions. Besides that I think most of the other changes are either non-existent or residual.

The policy runs aren't finished yet.

[Renato-Rodrigues]

@robertpietzcker

Since both the SSP2 scenarios and the 2040 paper scenarios are using the latest setup we tested, do I have the green light to proceed with merging this into the trunk with our current test configuration?

The current tested setup uses cm_seFeSectorShareDevMethod = sqSectorAvrgShare for all runs and cm_limitSolidsFossilRegi = EUR_regi for all non-calibration scenarios, including NPi and policy scenarios.

If you need any additional test runs, please let me know.

If approved, I'll update this PR to the current tested configuration and create a new PR to incorporate the cm_limitSolidsFossilRegi implementation.

The 2040 paper test runs are available at:
/p/projects/ecemf/REMIND/2040_scenarios/_mergeTest/remind_seFeSectorSharePenalization_2025/output

[johannah-pik]

Hi all :)
Its a bit hard to follow your discussions quickly.
Can someone summarize the implications of this fix for me?
In the transport reporting we read vm_demFeSector to get the fossil/syn/bio shares and apply a fix I discussed with @fschreyer to harmonize the shares for fepet and fedie as REMIND does not follow a reasonable mechanism to allocate them.

[Renato-Rodrigues]

Without the code in this PR, REMIND was free to allocate bio, syn, and fossil fuels across sectors.
If there is no additional sector specific cost to use a given carrier type of bio, syn or fossil origin, the model simply could choose randomly where it places more biofuels for example, making a sector emit more or less by pure chance.

To guide the model towards a more balanced distribution, the PR introduces a small monetary incentive (in the form of a cost penalization) to equalize the fuel shares across sectors. This incentive is designed to be minimal, so if there’s a sector-specific economic advantage to using a particular fuel type, the incentive should remain negligible.
In the absence of any other specific cost differences the shares of biofuels, synfuels, and fossil fuels in each sector should roughly match their overall share in the economy.

This adjustment takes effect from 2025 onward.
Post-merge, fedie and fepet should reflect bio, syn, and fossil fuel shares close to their overall proportions in secondary energy for these years.
If you intend to control these fuel use shares in transport based on some dynamics from edge-t, you would need to also include your logic at the REMIND equation side. Otherwise, if you simply use a equal share split between these carriers fuel origin types, you don't need anymore to do any post-processing as the split will be done endogenously by the model.

[Renato-Rodrigues]

the shares on years before 2025 shoudl be controlled by the q_shbiofe_up and q_shbiofe_lo equations.

[johannah-pik]

Thanks @Renato-Rodrigues.
this is very much appreciated. Currently, EDGE-T only sees liquids and a corresponding price for liquids.
Therefore, the share of fossil, syn- and biofuels is not influenced by the decision-making process in EDGE-T itself and remains on the REMIND side.

I will check once the fix is in place and remove the postprocessing step in our reporting.

[Renato-Rodrigues]

This PR is merging the changes discussed here in a disabled state.
Nothing will change from the REMIND results perspective.

A follow up PR will be created to include the default behavior change in REMIND, address the issue that motivated these code changes and point to test runs.