Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add data parser for Iceland #338

Merged
merged 7 commits into from
Feb 4, 2017
Merged

Add data parser for Iceland #338

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
8bf4f4e
Update README with some Windows setup troubleshooting
Spydarlee Jan 12, 2017
576fcd6
Remove unnecessary troubleshooting note from README
Spydarlee Jan 12, 2017
05f175f
Merge branch 'master' of https://github.com/corradio/electricitymap
Spydarlee Feb 1, 2017
2311478
Add data parser for Iceland
Spydarlee Feb 2, 2017
f46c14a
Add support for geothermal energy production
Spydarlee Feb 3, 2017
825695b
Iceland: set zero values for known unused energy sources
Spydarlee Feb 4, 2017
06ec40f
Iceland: Add capacity data and source
Spydarlee Feb 4, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ Each country has a GHG mass flow that depends on neighboring countries. In order
- Great Britain: [ENTSOE](https://transparency.entsoe.eu/content/static_content/Static%20content/web%20api/Guide.html)
- Greece: [ENTSOE](https://transparency.entsoe.eu/content/static_content/Static%20content/web%20api/Guide.html)
- Hungary: [ENTSOE](https://transparency.entsoe.eu/content/static_content/Static%20content/web%20api/Guide.html)
- Iceland [LANDSNET](http://amper.landsnet.is/MapData/api/measurements)
- Ireland: [ENTSOE](https://transparency.entsoe.eu/content/static_content/Static%20content/web%20api/Guide.html)
- Italy: [ENTSOE](https://transparency.entsoe.eu/content/static_content/Static%20content/web%20api/Guide.html)
- Latvia: [ENTSOE](https://transparency.entsoe.eu/content/static_content/Static%20content/web%20api/Guide.html)
Expand Down
4 changes: 3 additions & 1 deletion feeder/feeder.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
from bson.binary import Binary
from pymemcache.client.base import Client

from parsers import IS
from parsers import FR
from parsers import ENTSOE
from parsers import weather
Expand Down Expand Up @@ -105,6 +106,7 @@
'GR': ENTSOE.fetch_production,
'HU': ENTSOE.fetch_production,
'IE': ENTSOE.fetch_production,
'IS': IS.fetch_production,
'IT': ENTSOE.fetch_production,
'LT': ENTSOE.fetch_production,
'LU': ENTSOE.fetch_production,
Expand Down Expand Up @@ -300,7 +302,7 @@ def validate_production(obj, country_code):
raise Exception("Data from %s can't be in the future" % country_code)
if obj.get('production', {}).get('unknown', None) is None and \
obj.get('production', {}).get('coal', None) is None and \
country_code not in ['CH', 'NO']:
country_code not in ['CH', 'NO', 'IS']:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please remove from here. In principle, because you have "unknown" not None, this should not be a problem.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that I've added support for geothermal this is required again.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The purpose of this line is to make exceptions for specific countries that we want to show on the map even though they have no coal data.
However, having no coal data is different from having coal set to 0. For IS, if we can confirm that they do not produce any coal, we should set the coal key to 0 in the parser. We should do the same for all production types that we know IS doesn't have.

raise Exception("Coal or unknown production value is required for %s" % (country_code))
for k, v in obj['production'].iteritems():
if v is None: continue
Expand Down
11 changes: 7 additions & 4 deletions feeder/parsers/ENTSOE.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -271,13 +271,15 @@ def get_wind(values):
if 'Wind Onshore' in values or 'Wind Offshore' in values:
return values.get('Wind Onshore', 0) + values.get('Wind Offshore', 0)

def get_geothermal(values):
if 'Geothermal' in values:
return values.get('Geothermal', 0);

def get_unknown(values):
if 'Geothermal' in values \
or 'Marine' in values \
if 'Marine' in values \
or 'Other renewable' in values \
or 'Other' in values:
return values.get('Geothermal', 0) + \
values.get('Marine', 0) + \
return values.get('Marine', 0) + \
values.get('Other renewable', 0) + \
values.get('Other', 0)

Expand Down Expand Up @@ -335,6 +337,7 @@ def fetch_production(country_code, session=None):
'oil': get_oil(production_values),
'solar': production_values.get('Solar', None),
'wind': get_wind(production_values),
'geothermal': get_geothermal(production_values),
'unknown': get_unknown(production_values)
},
'storage': {
Expand Down
67 changes: 67 additions & 0 deletions feeder/parsers/IS.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
import requests
from datetime import datetime
from collections import defaultdict
import xml.etree.ElementTree as ET
import json

STATIONS = {

# http://www.landsvirkjun.com/company/powerstations/
'Bjarnaflag': 'geothermal', # No data?
'BLANDA_F': 'hydro',
'BUDAR_O': 'hydro',
'BURF_F': 'hydro',
'FLJOTSDA': 'hydro',
'Hafio': 'wind', # No data?
'HRAUN_F': 'hydro',
'IRAFOS_F': 'hydro',
'KRAFLA_F': 'geothermal',
'LAXA_F': 'hydro',
'LAXARVAT': 'hydro',
'laxa': 'hydro', # No data?
'LJOSIF_F': 'hydro',
'SIG_F': 'hydro',
'Steingrimsstod': 'hydro', # No data?
'SULTAR_F': 'hydro',
'VATNSHAM': 'hydro',

# https://en.wikipedia.org/wiki/List_of_power_stations_in_Iceland
'KOLVID': 'geothermal',
'LAGARF': 'hydro',
'MJOLKA': 'hydro',
'REY': 'geothermal',
'X_NESJAV': 'geothermal',
'SVA': 'geothermal',
}

def fetch_production(country_code='IS', session=None):

# Query Landsnet for latest power production data for Iceland
r = session or requests.session()
url = 'http://amper.landsnet.is/MapData/api/measurements'
response = r.get(url)
json_obj = json.loads(response.text)

production = defaultdict(float)

# Calculate production values for each power station
for key, value in STATIONS.items():
items = [item for item in json_obj if item["substation"] == key and item["MW"] >= 0]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what happens to all stations that are not matched? They are simply ignored?
We should add them to unknown maybe?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They are ignored, yeah. I'm not sure it's a good idea to funnel them all into unknown by default because I don't think everything we get through that API is actually power generation.

For example, there is a substation BRENNIME which Google translates/expands to Brennimelur which seems to be a capacitor bank: http://www.rst.is/en/verkefni/2001/BRE-30.html not a generator/power station.

The data for that example has some pretty big MW values, so it'd skew our results quite heavily. Without better knowledge of Icelandic I was thinking it might be best for now to just stick with the stations that @birkir made specific note of? What do you think?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok I agree! Maybe add a small comment explaining that in the code.

mw = sum(item['MW'] for item in items)
production[value] = production[value] + mw;

# Get datetime for last update (e.g. 2017-02-02T14:35:00)
totalpowerflow = next(item for item in json_obj if item["key"] == "TOTAL_POWER_FLOW")
datetime_last = datetime.strptime(totalpowerflow["time"], '%Y-%m-%dT%H:%M:%S')

data = {
'countryCode': country_code,
'production': dict(production),
'datetime': datetime_last,
'storage': {},
'source': 'landsnet.is',
}
return data

if __name__ == '__main__':
print fetch_production()
1 change: 1 addition & 0 deletions shared/co2eq_parameters.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ defaultCo2eqFootprint = {
'oil': 650,
'solar': 45,
'wind': 12,
'geothermal': 24,
'unknown': 700, // assume conventional
'other': 700 // same as 'unknown'. Here for backward compatibility
}; // in gCo2eq/kWh
Expand Down
2 changes: 2 additions & 0 deletions web/app/countrytable.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ function CountryTable(selector, co2Color) {
'solar': '#f27406',
'hydro': '#2772b2',
'biomass': '#166a57',
'geothermal': 'yellow',
'nuclear': '#AEB800',
'gas': '#bb2f51',
'coal': '#ac8c35',
Expand All @@ -42,6 +43,7 @@ function CountryTable(selector, co2Color) {
'solar',
'hydro',
'hydro storage',
'geothermal',
'biomass',
'nuclear',
'gas',
Expand Down