Split Italy parser

[Raffox97]

If you look on the ENTSOE site, Italy is split in big regions: the north, the centre-north, the centre-south, the south and the islands. It could be a good idea to have that split also here on the map.

[alixunderplatz]

@Raffox97 Power flows between the zones exist, the only problem are the installed generation capacities by type, which are not filled on ENTSO-E if you look at the bidding zones. These only exist for Italy as a whole there. Finding out how much is installed in what zone will be really tough. As far as I remember, the Italian system operator TERNA was summarizing the fossil types as "thermal" without splitting it to gas and coal.
Are you from Italy? Maybe you can find out about the installed capacity by type of each zone somewhere?

[brunolajoie]

Hey guys! Great idea. I personally think that capacity installed is of much less importance than real time data (an can often be found later on), so if you wish to split italy in big regions and if the real data time data is robust, feel free to go! But we'll need to make sure that real time data is at least as good as Italy as a whole! Thanks!

[Raffox97]

Perfect, unluckly I can't find a good site where to see the division, but the hope is the last to die, so if there are any news, I'll post it in this issue.

[corradio]

Have you looked at ENTSOE? I know they do the breakdown. The question whether or not the data is accurate/complete enough:
https://transparency.entsoe.eu/generation/r2/actualGenerationPerProductionType/show

[Raffox97]

What about dash lines to divide regions?

[corradio]

Here are the steps to move forward:

• verify that per-zone generation breakdown properly sums up to country data (using ENTSOE)
• identify geometries to be used (see ENTSOE geojson bidding zones #917)
• add/update exchanges in exchanges.json, and zones in zones.json

[alixunderplatz]

Italy with all these exchanges and seperate zones will look beautiful once this split is done!

I had a look at the data and here are the encouraging results:

1. The sum of per zone-generation is equal to the total generation ✔️
2. geometries of the zones luckily follow borders of administrative regions. ✔️ 👍
   Some cities/very small zones in Italy South and Sicily seem to have their own bidding zones, so we should think about summarizing them with their surrounding bigger zone and dismiss the "exchanges" with the respective main zone. The larger bidding zones coincide with the "region" attributes of the Natural Earth shapefiles (see image for mapping regions to bidding zones).

Here is a geographical breakdown:

3. These are the exchanges from the bidding zones that must be implemented (no duplicates in the list):

Use these links as support with the map above:

ENTSO-E map
ENTSO-E exchanges for IT bidding zones

These first four are equal to the exchanges from IT to the northern neighbours as of now.

• IT-NO <-> FR
• IT-NO <-> CH
• IT-NO <-> AT (or DE-AT-LU bidding zone)
• IT-NO <-> SI

---

• IT-CNO <-> IT-NO
• IT-CNO <-> IT-CSO

---

• IT-SACODC <-> IT-CNO

• IT-SACODC <-> IT-SAR
  These two interconnections are a bit special. The first connects both Corsica and Sardinia with Italy North. See Wikipedia.
  The difference of the first and second exchange is the amount power that is leaving the HVDC line on FR-COR (up to 50 MW). This needs to be examined a bit deeper in a seperate investigation, because of the different directions of the power flows.

• IT-SAR <-> IT-SACOAC - See last sentence of this Wikipedia site - that's a seperate AC connection linking IT-SAR and FR-COR.

• IT-SAR <-> IT-CSO

---

• IT-CSO <-> IT-SO

I suggest including the generation of the small bidding zones IT-FO, IT-BR, IT-RO in IT-SO. Else, some of the small areas surrounding these cities could be seperated, but from my point this is unnecessary.
Maybe even Sicily and South should be merged into one zone, because I am not sure about the correctness of given power flows between them.

• from IT-RO there is one link between Sicily and Rossano (within IT-South), so exchange
• IT-RO <-> IT-SIC will become IT-SO <-> IT-SIC, but there are other power lines from the western coast of IT-South leading to Sicily. So I have some concerns this data covers the total exchange.
• IT-BR is the substation linking Greece with South, so exchange
• IT-BR <-> GR (or IT <-> GR as of now) will become IT-SO <-> GR, if IT-BR generation will be included in IT-SO

---

• IT-PR generation can be included in IT-SIC

• IT-SIC - MT

---

• The link between Italy and Montenegro (ME) will go online within the next year or so, but someone will probably catch that from the news.

[alixunderplatz]

Breakdown of the "region" and "region_cod" attributes in Natural Earth to the zones for geometries:

Italy North IT-NO:
['Emilia-Romagna', 'Friuli-Venezia Giulia', 'Liguria', 'Lombardia',
'Piemonte', 'Trentino-Alto Adige', 'Valle d'Aosta', 'Veneto']
or
['IT-45', 'IT-36', 'IT-42', 'IT-25', 'IT-21', 'IT-32', 'IT-23', 'IT-34']

Italy Central North IT-CNO:
['Marche', 'Toscana', 'Umbria']
or
['IT-57', 'IT-52', 'IT-55']

Italy Central South IT-CSO:
['Abruzzo', 'Campania', 'Lazio']
or
['IT-65', 'IT-72', 'IT-62']

Italy South IT-SO
['Apulia', 'Basilicata', 'Calabria', 'Molise']
or
['IT-75', 'IT-77', 'IT-78', 'IT-67']

Italy-Sicily IT-SIC
['Sicily']
or
['IT-82']

Italy-Sardinia IT-SAR
['Sardegna']
or
['IT-88']

[alixunderplatz]

Another summary of the next steps with some updates of what I had written above:

• decide, whether to include generation for bidding zones Foggia, Brindisi and Rossano in IT-SO and Priolo in IT-SIC

  • the simple list of "regions" would require a breakdown to adm_1 or similar attributes on county level - there are over 100 in Italy. Not a problem, but just some more work to do.
  • In addition, these micro-bidding zones are just valid for single substations/nodes/power plants that suffer some constraints (due to the grid or sth similar). They export most of their generation to the surrounding zones anyway.
  • As opposed to what I wrote above for Brindisi, the connection to GR is not starting there, but in the substation "Galatina", which has 2 connections - one from Brindisi and one from Taranto. I find this highly confusing / inconsistent. Either the map i posted above with the bidding zones is not up-to-date anymore, or the interconnections are named improperly. However, the IT->GR exchange has the same values as IT-Brindisi -> GR. This would also encourage including IT-Brindisi in IT-South.
    See excerpt fromENTSO-E map:
    

• decide, whether to merge generation of Sicily (IT-SIC) and South (IT-SO) into one common zone

  • exchanges between IT-SO and IT-SIC might be incomplete. There is Rossano <-> Sicily, but no seperate South <-> Sicily interconnection given on ENTSO-E, although there is another power line on the western coast. Again, either the map i posted above with the bidding zones is not up-to-date anymore, or the interconnections are named improperly.
    See excerpt from ENTSO-E map:
    

• make sure that Rossano -> Sicily represents the entire exchange between IT-South and IT-Sicily or if there is something missing on the ENTSO-E platform

• Unfortunately, the exchange flow map on ENTSO-E is not working (at least on my PC) at the moment, so there is no option for another visual check of the bidding zones. Maybe someone can help finding the recent borders of them and double check?

• add/update exchanges in exchanges.json, and zones in zones.json

[corradio]

As mentioned by @tmslaine on https://github.com/tmrowco/electricitymap/issues/917

For Italy http://www.mercatoelettrico.org/En/mercati/MercatoElettrico/Zone.aspx has a list of how Italian administrative regions are split into bidding zones.

Any roadblocks for getting started?

[alixunderplatz]

@corradio First off, we have to decide whether to include the "micro-bidding zones" in the larger ones or not. Usually, they just contain a larger generator that is operating under output restrictions due to the grid or sth like that. So from my point of view, they can be aggregated with their larger surrounding zone without problems. Administrative regions are a bit too large for each of these zones.

I think the exchanges represent the total exchange between the larger bidding zones (not just from one substation to another, but the sum/balance of all exchanges - this got me confused/worried a bit earlier), so we're also good to go here.

---

• create new geometries based on the breakdown above

• include new zones in parser and summarize their generation if necessary as mentioned above. Luckily, the large zones follow administrative borders.

  • IT-NO
  • IT-CNO
  • IT-CSO
  • IT-SO (incl. IT-FO, IT-BR, IT-RO / Foggia, Brindisi and Rossano)
  • IT-SIC (incl. IT-PR / Priolo)
  • IT-SAR

• create exchanges with neighbouring countries <-> IT-...

• create exchanges between IT zones

• include price data for the zones

Feel free to edit this list directly if something is missing.
Catch me on Slack to discuss details if anything is unclear (especially with interconnectors) ;)

[antifa-ev]

See my pull request:
#1536

[jarek]

Copying out my message from Slack for the record. Steps to ideally create new geometries would be:

• look at the Natural Earth shapefile (linked in Add Italy bidding zones (GeoJSON) #1536 (comment)) and note region_cod for each of the provinces in Split Italy parser #1029 (comment)
• edit topogen.sh to include Italy as country that is broken down
• edit generate-geometries.js to allow selecting by region_cod (someone can help with this)
• edit generate-geometries.js around line 305 to change Italy from being just id:'ITA' to being split, probably similar to how India is split

[chazanov]

I would like to further work on it, but I've got some problems with my local electricitymap instance. When doing this pull request, I've had not yet been able to run it locally.

Now docker works for me, but the colors are wrong and Japan is missing, despite I've just cloned from git.

[corradio]

Hi @chazanov, It makes sense that the colors are wrong (we use fake data locally). Japan is missing because this PR is behind master. Don't worry about it, it will be fixed once merged. I suggest you sync up with @antifa-ev on Slack (slack.tmrow.com) to make sure there's no double work. Olivier

…

On Wed, Aug 15, 2018 at 7:18 PM chazanov ***@***.***> wrote: I would like to further work on it, but I've got some problems with my local electricitymap instance. When doing this pull request, I've had not yet been able to run it locally. Now docker works for me, but the colors are wrong and Japan is missing, despite I've just cloned from git. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1029 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABlEKBzevGpMR8LxOuLJClc8hbVVBC9Vks5uRFfggaJpZM4Ru0t4> .

[corradio]

@alixunderplatz I need help on making the mapping between ENTSOE bidding zones and the zones that you defined.

Can you help?

[alixunderplatz]

@corradio sure! :)

IT-NO" includes ENTSO-E zone "BZN|IT-North"

"IT-CNO" includes ENTSO-E zone "BZN|IT-Centre-North"

"IT-CSO" includes ENTSO-E zone "BZN|IT-Centre-South"

"IT-SO" includes the sum of ENTSO-E zones "BZN|IT-South", "BZN|IT-Brindisi", "BZN|IT-Foggia", "BZN|IT-Rossano"

"IT-SAR" includes ENTSO-E zone "BZN|IT-Sardinia"

"IT-SIC"includes the sum of ENTSO-E zones "BZN|IT-Sicily" and "BZN|IT-Priolo"

I hope this helps :) Reach out if something is unclear with the national power flows between these zones

[corradio]

@alixunderplatz thanks! And what is the reason for not keeping a 1:1 mapping between our zones and ENTSOE BZNs for SIC and SO?

[alixunderplatz]

@corradio We include these "micro bidding zones", which consist of a power plant feeding in to one node in the grid, in the surrounding larger bidding zones.
Imagine there is a substation in the grid where one power plant and its units are feeding in, and that's it. These micro-zones only do exist due to grid or market restrictions.

[corradio]

very clear thanks.

[corradio]

Should be online in a bit. The only thing missing now is SACODC/AC.

[alixunderplatz]

I totally forgot about generation capacities for the new regions.
This website from TERNA has a very nice breakdown by regions for renewable sources, which have to be aggregated in the respective bidding zones:

https://www.terna.it/en-gb/sistemaelettrico/fontirinnovabili.aspx

---

ENTSO-E only has capacities for the entire country, which seem to randomly jump up and down in some categories every year (compare oil / other / hard coal for the last years):

No data for bidding zones:

But there is a breakdown per generation unit per bidding zone, which we could check and use if it is complete:

[alixunderplatz]

Here we still need to figure out how to deal with the complex exchange structrure between Italy Central North <-> Corsica / Sardinia <-> Italy Central South zones.
Maybe someone can take another look on the capacities per region (at least for the renewables)
Otherwise we could close this issue since the split was merged already.

[corradio]

@alixunderplatz would you have time to help a bit here?

[alixunderplatz]

@corradio I'll look into this from 3rd October onwards ;)
entsoe data plus EDF data for Corse will help getting the last bits done here :)

[alixunderplatz]

By the way: the interconnector between Italy/Italy south and Greece still isn't working on the map
Somebody could check the "regional codes"/ bidding zone codes on entsoe for this interconnector for changes

[alixunderplatz]

I'm currently reviewing the data of the DC-link between:
Central North Italy (IT-CNO)<->Corsica(FR-COR)<->Sardinia(IT-SAR).

Note: At the moment the DC-link seems to be out of service or the data is N/A and we probably cannot reproduce/test the power flows in real time on the map! 😭

Basically, the DC interconnector has a total capacity of 300 MW and can work in both directions. Capacity of the converter on Corsica is 50 MW.

1. It seems that Corsica doesn't export any energy into this DC-link and only is importing energy from it with a maximum of 50 MW.
2. What worried me in the past is the existing short AC-link between Corsica and Sardinia.
   However: Corsica is (mostly) a constant net importer of electricity via the AC-link from Sardinia, based on the 2019 data of ENTSOE. There have been only 22 hours so far in 2019 (of available data), where Corsica exported energy to Sardinia via the 150kV AC-cable, with a maximum magnitude of 12 MW and a total energy of only 67 MWh this year, which is insignificant compared to the production power and total production on Sardinia (<=1%).

Both 1.+2. leads to the assumption/result, that the carbon intensity on Corsica doesn't influence the intensity of Sardinia or Central North Italy, because energy is (almost exclusively!) imported to Corsica 👍 That will make things easier without any real-time production data for Corsica :)

I made some calculations with the data given on ENTSOE and was able to get some valid values out of it. This is very promising! :)

DC-link flow from Central North Italy towards Sardinia
Central North is exporting to DC-link, Corsica is importing from the DC-link, the difference is imported by Sardinia
DC-link flow from Sardinia towards Central North
Sardinia is exporting to DC-link, Corsica is importing from the DC-link, the difference is imported by Central North Italy
The difference between orange and yellow and blue and grey is what is imported by Corsica (max 50 MW, valid!).

Seperate AC-link Sardinia<->Corsica

Calculated flow between IT-CNO <-> IT-SAR via DC-link

Calculated flow from IT-CNO to FR-COR and from IT-SAR to FR-COR

Calculated total flow from IT-SAR to FR-COR via DC-link plus AC-cable

I'll post a short transcription of the logic behind the calculations and which data from entsoe we need in the next comment and will attach the .xlsx file with my calculations. Then we should be able to get these interconnectors on the map!!!

👍👍👍

[alixunderplatz]

The underlying ENTSOE physical flow data is from here.

The Excel-sheet with my calculations and the graphs from above is attached here:
SARDINIA_edit.xlsx

These interconnectors on ENTSOE are required:
BZN|IT-Centre-North - BZN|IT-SACODC (= IT-CNO <-> SACODC)
BZN|IT-SACOAC - BZN|IT-Sardinia (= FR-COR <-> IT-SAR via AC cable)
BZN|IT-SACODC - BZN|IT-Sardinia (SACODC <-> IT-SAR)
SACOAC = exchange between FR-COR and IT-SAR via an AC-connection
SACODC = exchange with the connected zones of the DC-connection between IT-CNO,FR-COR,IT-SAR

Basically, the following stuff is the logic behind the calculations. The "else = 0" logic is only used in my "unidirectional" data table in the excel sheet and could be dismissed when coding. Actually, behind "else" the code for the opposite direction of the respective flow should follow (with a negative value as a result to flip the direction on the map)...

IT-CNO exports to IT-SAR

if "SACODC to IT-SAR" <= "IT-CNO to SACODC"
then flow = "SACODC to IT-SAR"
else = 0

IT-CNO exports to FR-COR

if "IT-CNO to SACODC" > "SACODC to IT-SAR"
then flow = "IT-CNO to SACODC" - "SACODC to IT-SAR"
else = 0

IT-SAR exports via SACODC to FR-COR

if "IT-SAR to SACODC" > 0
then flow = "IT-SAR to SACODC" - "SACODC to IT-CNO"
else = 0

FR-COR exports (via SACODC) to IT-SAR (never happens?!)

if "SACODC to IT-SAR" > "IT-CNO to SACODC"
then flow = "SACODC to IT-SAR" - "IT-CNO to SACODC"
else = 0

IT-SAR exports to IT-CNO

if "SACODC->IT-CNO" - "IT-SAR->SACODC" <= 0
then flow = "SACODC->IT-CNO"
else = 0

FR-COR exports to IT-CNO (never happens?!)

if "SACODC->IT-CNO" - "IT-SAR->SACODC" < 0
then flow = "SACODC->IT-CNO" - "IT-SAR->SACODC"
else = 0

Direction of IT-SAR <-> FR-COR exchange with via AC- and DC cables (total balance)

flow = "IT-SAR via SACODC to FR-COR" + "IT-SAR via SACOAC to FR-COR" - "FR-COR via SACOAC to IT-SAR" - "FR-COR via SACODC to IT-SAR"
If this balance is positive, flow is from IT-SAR to FR-COR,
If this balance is negative, flow is from FR-COR to IT-SAR.

I hope I haven't messed things up. Couldn't find a way to describe it easier, so please forgive me^^
There are many N/A values in the underlying data charts from entsoe. Does this require any additional special handling?
Reach out to me here or on Slack if you have any troubles understanding this mess^^

[PETILLON-Sebastien]

Looks pretty accurate to me. Is there something missing ?
Question : this would work for historical data as well as real-time, am I wrong ?
(Yes I need it for my lovely FR_O parser 😍 )

[corradio]

@alixunderplatz is there any concrete action items we can do to close this one off?
Now that @robertahunt joined Tomorrow, maybe she can help in adding the missing exchanges?

[alixunderplatz]

@corradio I think my comments from 3rd October 2019 contain what needs to be done in the code.
@robertahunt Congratulations on joining tomorrow! 😃 👍 Feel free to reach out on Slack regarding any questions about the interconnectors between Italy/Corsica/Sardinia. Maybe it's the best to write a seperate parser outside the ENTSOE.py for fetching these weird exchanges.
Basically, if you study the formulas in the excel-sheet from above, you'll see the logic behind directions and assignments of flows.

The following three arrows/exchanges must be added to the map:
1.) IT-CNO <-> IT-SAR
2.) IT-CNO <-> FR-COR (it's already there, seems to be correct for IT-CNO->FR-COR)
3.) IT-SAR <-> FR-COR (summing up DC-flow from SAR to COR + AC-flow between SAR and COR.)

N/As in the ENTSOE-data kind of indicate when the DC-interconnector is out of service, so only the AC-flow between SAR and COR ("SACOAC") will continue in that case.

[robertahunt]

@alixunderplatz
Thanks for the warm welcome! I have a couple burning tasks to take care of at the moment, but this is on my radar and I will be sure to approach you for some advice when I get closer.
Cheers,
Roberta

[corradio]

Closing in favour of #2991