Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Creates HERON/data directory to store common HERON data. #351

Merged
merged 9 commits into from
Apr 9, 2024
File renamed without changes.
File renamed without changes.
File renamed without changes.
9 changes: 9 additions & 0 deletions data/orig_locations.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
renamed: tests/integration_tests/TSA/Sine/arma.pk -> data/TSA/Sine/arma.pk
renamed: tests/integration_tests/TSA/NYISO/nyiso_arma_2yr.pk -> data/TSA/NYISO/nyiso_arma_2yr.pk
renamed: tests/integration_tests/TSA/Sine30yr/arma_30yr.pk -> data/TSA/Sine30yr/arma_30yr.pk
renamed: tests/integration_tests/ARMA/Sine/arma.pk -> data/ARMA/Sine/arma.pk
renamed: tests/integration_tests/ROM/linear_rom.pk -> data/ROM/linear_rom.pk
renamed: tests/integration_tests/TSA/Sine_Hour/arma.pk -> data/TSA/Sine_Hour/arma.pk
renamed: tests/integration_tests/XML_check/transfers.py -> data/XML_check/transfers.py
renamed: tests/integration_tests/mechanics/static_history/Static.csv -> data/mechanics/static_history/Static.csv
renamed: tests/integration_tests/mechanics/static_history/Static.csv -> data/mechanics/static_history/Static.csv
16 changes: 16 additions & 0 deletions src/Placeholders.py
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this file, it might be worth updating the InputData descriptions within the get_input_specs( ) methods of the ARMA(Placeholder) and Function(Placeholder) classes to talk about using %HERON_Data% instead of/in addition to %HERON%. In the modified files, these are lines 169 and 264 respectively (could not find a way to link to them directly in this comment).

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, I added descriptions of HERON_DATA to those get_input_specs.

Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,20 @@ def read_input(self, xml):
# magic word for "relative to FARM root"
farm_path = hutils.get_farm_loc()
self._target_file = os.path.abspath(self._source.replace('%FARM%', farm_path))
elif self._source.startswith('%HERON_DATA%'):
#magic word to try and find HERON data files
#note that this first checks for a HERON_DATA environment variable
heron_data = os.environ.get("HERON_DATA")
if heron_data is None:
heron_path = hutils.get_heron_loc()
possible_path = os.path.join(heron_path,"data")
if os.path.exists(possible_path):
heron_data = possible_path
else:
self.raiseAnError(IOError,"ERROR trying to find %HERON_DATA% and not found at", possible_path, "and no HERON_DATA environmental variable found")
elif not os.path.exists(heron_data):
self.raiseAnError(IOError, "ERROR path", heron_data, "found in HERON_DATA environment variable does not seem to exist")
self._target_file = os.path.abspath(self._source.replace('%HERON_DATA%', heron_data))
else:
# check absolute path
rel_interp = os.path.abspath(os.path.join(self._workingDir, self._source))
Expand Down Expand Up @@ -152,6 +166,7 @@ def get_input_specs(cls):
The RAVEN ARMA ROM should be trained and serialized before using it in HERON. The text
of this node indicates the location of the serialized ROM. This location is usually relative
with respect to the HERON XML input file; however, a full absolute path can be used,
or the path can be prepended with ``\%HERON_DATA\%'' to be relative to the HERON data directory (``HERON/data'' in the git repository, but the location of ``\%HERON_DATA\%'' can be overridden by exporting a ``HERON_DATA'' environmental variable),
or the path can be prepended with ``\%HERON\%'' to be relative to the installation
directory of HERON.""")
specs.addParam('name', param_type=InputTypes.StringType, required=True,
Expand Down Expand Up @@ -247,6 +262,7 @@ def get_input_specs(cls):
Python functions have access to the variables within the dispatcher. The text
of this node indicates the location of the python file. This location is usually relative
with respect to the HERON XML input file; however, a full absolute path can be used,
or the path can be prepended with ``\%HERON_DATA\%'' to be relative to the HERON data directory (``HERON/data'' in the git repository, but the location of ``\%HERON_DATA\%'' can be overridden by exporting a ``HERON_DATA'' environmental variable),
or the path can be prepended with ``\%HERON\%'' to be relative to the installation
directory of HERON.""")
specs.addParam('name', param_type=InputTypes.StringType, required=True,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -113,7 +113,7 @@
<Files>
<Input name="inner_workflow" type="raven">../inner.xml</Input>
<Input name="heron_lib">../heron.lib</Input>
<Input name="transfers">../%BASE_WORKING_DIR%/../../transfers.py</Input>
<Input name="transfers">../%HERON_DATA%/XML_check/transfers.py</Input>
</Files>

<OutStreams>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -113,7 +113,7 @@
<Files>
<Input name="inner_workflow" type="raven">../inner.xml</Input>
<Input name="heron_lib">../heron.lib</Input>
<Input name="transfers">../%BASE_WORKING_DIR%/../../transfers.py</Input>
<Input name="transfers">../%HERON_DATA%/XML_check/transfers.py</Input>
</Files>

<OutStreams>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -120,8 +120,8 @@
</Components>

<DataGenerators>
<!-- <ARMA name='Price' variable="Signal">%BASE_WORKING_DIR%/../../../TSA/Sine/arma.pk</ARMA> -->
<Function name="transfers">%BASE_WORKING_DIR%/../../transfers.py</Function>
<!-- <ARMA name='Price' variable="Signal">%HERON_DATA%/TSA/Sine/arma.pk</ARMA> -->
<Function name="transfers">%HERON_DATA%/XML_check/transfers.py</Function>
</DataGenerators>

</HERON>
Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,6 @@
</Components>

<DataGenerators>
<ARMA name='flex' variable="Signal">%BASE_WORKING_DIR%/../../../TSA/Sine/arma.pk</ARMA>
<ARMA name='flex' variable="Signal">%HERON_DATA%/TSA/Sine/arma.pk</ARMA>
</DataGenerators>
</HERON>
Original file line number Diff line number Diff line change
Expand Up @@ -120,8 +120,8 @@
</Components>

<DataGenerators>
<!-- <ARMA name='Price' variable="Signal">%BASE_WORKING_DIR%/../../../TSA/Sine/arma.pk</ARMA> -->
<Function name="transfers">%BASE_WORKING_DIR%/../../transfers.py</Function>
<!-- <ARMA name='Price' variable="Signal">%HERON_DATA%/TSA/Sine/arma.pk</ARMA> -->
<Function name="transfers">%HERON_DATA%/XML_check/transfers.py</Function>
</DataGenerators>

</HERON>
2 changes: 1 addition & 1 deletion tests/integration_tests/cluster_check/heron_input.xml
Original file line number Diff line number Diff line change
Expand Up @@ -110,7 +110,7 @@
</Components>

<DataGenerators>
<ARMA name='Price' variable="Signal">%BASE_WORKING_DIR%/../../ARMA/Sine/arma.pk</ARMA>
<ARMA name='Price' variable="Signal">%HERON_DATA%/ARMA/Sine/arma.pk</ARMA>
<Function name="transfers">transfers.py</Function>
</DataGenerators>

Expand Down
4 changes: 2 additions & 2 deletions tests/integration_tests/mechanics/ROM_source/heron_input.xml
Original file line number Diff line number Diff line change
Expand Up @@ -71,8 +71,8 @@
</Components>

<DataGenerators>
<ARMA name='placeholder' variable="Signal">%BASE_WORKING_DIR%/../../../TSA/Sine/arma.pk</ARMA>
<ROM name="myROM">%BASE_WORKING_DIR%/../../../ROM/linear_rom.pk</ROM>
<ARMA name='placeholder' variable="Signal">%HERON_DATA%/TSA/Sine/arma.pk</ARMA>
<ROM name="myROM">%HERON_DATA%/ROM/linear_rom.pk</ROM>
</DataGenerators>

</HERON>
Original file line number Diff line number Diff line change
Expand Up @@ -159,7 +159,7 @@


<DataGenerators>
<CSV name='synth' variable="Signal">%BASE_WORKING_DIR%/../../Static.csv</CSV>
<CSV name='synth' variable="Signal">%HERON_DATA%/mechanics/capacity_factors/Static.csv</CSV>
</DataGenerators>

</HERON>
Original file line number Diff line number Diff line change
Expand Up @@ -161,7 +161,7 @@


<DataGenerators>
<CSV name='synth' variable="Signal">%BASE_WORKING_DIR%/../../../mechanics/capacity_factors/Static.csv</CSV>
<CSV name='synth' variable="Signal">%HERON_DATA%/mechanics/capacity_factors/Static.csv</CSV>
</DataGenerators>

</HERON>
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,7 @@
</Components>

<DataGenerators>
<ARMA name='flex' variable="Signal">%BASE_WORKING_DIR%/../../../TSA/Sine30yr/arma_30yr.pk</ARMA>
<ARMA name='flex' variable="Signal">%HERON_DATA%/TSA/Sine30yr/arma_30yr.pk</ARMA>
<Function name="functions">functions.py</Function>
</DataGenerators>
</HERON>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,7 @@
</Components>

<DataGenerators>
<ARMA name='Price' variable="Signal">%BASE_WORKING_DIR%/../../../TSA/Sine/arma.pk</ARMA>
<ARMA name='Price' variable="Signal">%HERON_DATA%/TSA/Sine/arma.pk</ARMA>
<Function name="transfers">transfers.py</Function>
</DataGenerators>

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,7 @@
</Components>

<DataGenerators>
<ARMA name='flex' variable="Signal">%BASE_WORKING_DIR%/../../../TSA/Sine30yr/arma_30yr.pk</ARMA>
<ARMA name='flex' variable="Signal">%HERON_DATA%/TSA/Sine30yr/arma_30yr.pk</ARMA>
</DataGenerators>
</HERON>

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@
</Components>

<DataGenerators>
<ARMA name='flex' variable="Signal">%BASE_WORKING_DIR%/../../../TSA/Sine30yr/arma_30yr.pk</ARMA>
<ARMA name='flex' variable="Signal">%HERON_DATA%/TSA/Sine30yr/arma_30yr.pk</ARMA>
<Function name="functions">functions.py</Function>
</DataGenerators>
</HERON>
Expand Down
2 changes: 1 addition & 1 deletion tests/integration_tests/mechanics/labels/heron_input.xml
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,7 @@
</Components>

<DataGenerators>
<ARMA name='Price' variable="Signal">%BASE_WORKING_DIR%/../../../TSA/Sine_Hour/arma.pk</ARMA>
<ARMA name='Price' variable="Signal">%HERON_DATA%/TSA/Sine_Hour/arma.pk</ARMA>
<Function name="transfers">transfers.py</Function>
</DataGenerators>

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,6 @@
</Components>

<DataGenerators>
<ARMA name='flex' variable="Signal">%BASE_WORKING_DIR%/../../../../TSA/Sine30yr/arma_30yr.pk</ARMA>
<ARMA name='flex' variable="Signal">%HERON_DATA%/TSA/Sine30yr/arma_30yr.pk</ARMA>
</DataGenerators>
</HERON>
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,6 @@
</Components>

<DataGenerators>
<ARMA name='flex' variable="Signal">%BASE_WORKING_DIR%/../../../../TSA/Sine/arma.pk</ARMA>
<ARMA name='flex' variable="Signal">%HERON_DATA%/TSA/Sine/arma.pk</ARMA>
</DataGenerators>
</HERON>
Original file line number Diff line number Diff line change
Expand Up @@ -122,6 +122,6 @@
</Components>

<DataGenerators>
<ARMA name='flex' variable="Signal">%BASE_WORKING_DIR%/../../../../TSA/Sine/arma.pk</ARMA>
<ARMA name='flex' variable="Signal">%HERON_DATA%/TSA/Sine/arma.pk</ARMA>
</DataGenerators>
</HERON>
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,6 @@
</Components>

<DataGenerators>
<ARMA name='flex' variable="Signal">%BASE_WORKING_DIR%/../../../../TSA/Sine/arma.pk</ARMA>
<ARMA name='flex' variable="Signal">%HERON_DATA%/TSA/Sine/arma.pk</ARMA>
</DataGenerators>
</HERON>
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,6 @@
</Components>

<DataGenerators>
<ARMA name='flex' variable="Signal">%BASE_WORKING_DIR%/../../../../TSA/Sine/arma.pk</ARMA>
<ARMA name='flex' variable="Signal">%HERON_DATA%/TSA/Sine/arma.pk</ARMA>
</DataGenerators>
</HERON>
Original file line number Diff line number Diff line change
Expand Up @@ -123,6 +123,6 @@
</Components>

<DataGenerators>
<ARMA name='flex' variable="Signal">%BASE_WORKING_DIR%/../../../../TSA/Sine/arma.pk</ARMA>
<ARMA name='flex' variable="Signal">%HERON_DATA%/TSA/Sine/arma.pk</ARMA>
</DataGenerators>
</HERON>
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,7 @@
</Components>

<DataGenerators>
<ARMA name='flex' variable="Signal">%BASE_WORKING_DIR%/../../../TSA/Sine/arma.pk</ARMA>
<ARMA name='flex' variable="Signal">%HERON_DATA%/TSA/Sine/arma.pk</ARMA>
<Function name="transfers">transfers.py</Function>
</DataGenerators>
</HERON>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,7 @@
</Components>

<DataGenerators>
<ARMA name='Speed' variable="Signal">%BASE_WORKING_DIR%/../../../TSA/Sine/arma.pk</ARMA>
<ARMA name='Speed' variable="Signal">%HERON_DATA%/TSA/Sine/arma.pk</ARMA>
<Function name="transfers">transfers.py</Function> <!-- TODO specify the variables needed? -->
</DataGenerators>

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,7 @@
</Components>

<DataGenerators>
<ARMA name='flex' variable="Signal">%BASE_WORKING_DIR%/../../../TSA/Sine30yr/arma_30yr.pk</ARMA>
<ARMA name='flex' variable="Signal">%HERON_DATA%/TSA/Sine30yr/arma_30yr.pk</ARMA>
<Function name="functions">functions.py</Function>
</DataGenerators>
</HERON>
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,7 @@
</Components>

<DataGenerators>
<ARMA name='flex' variable="Signal">%BASE_WORKING_DIR%/../../../TSA/Sine30yr/arma_30yr.pk</ARMA>
<ARMA name='flex' variable="Signal">%HERON_DATA%/TSA/Sine30yr/arma_30yr.pk</ARMA>
<Function name="functions">functions.py</Function>
</DataGenerators>
</HERON>
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,6 @@
</Components>

<DataGenerators>
<ARMA name='flex' variable="Signal">%BASE_WORKING_DIR%/../../../TSA/Sine30yr/arma_30yr.pk</ARMA>
<ARMA name='flex' variable="Signal">%HERON_DATA%/TSA/Sine30yr/arma_30yr.pk</ARMA>
</DataGenerators>
</HERON>
Original file line number Diff line number Diff line change
Expand Up @@ -122,7 +122,7 @@
</Components>

<DataGenerators>
<ARMA name='Price' variable="Signal">%BASE_WORKING_DIR%/../../../TSA/Sine/arma.pk</ARMA>
<ARMA name='Price' variable="Signal">%HERON_DATA%/TSA/Sine/arma.pk</ARMA>
<Function name="transfers">transfers.py</Function>
</DataGenerators>

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,7 @@
</Components>

<DataGenerators>
<ARMA name='Price' variable="Signal">%BASE_WORKING_DIR%/../../../TSA/Sine/arma.pk</ARMA>
<ARMA name='Price' variable="Signal">%HERON_DATA%/TSA/Sine/arma.pk</ARMA>
<Function name="transfers">transfers.py</Function>
</DataGenerators>

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -146,7 +146,7 @@
</Components>

<DataGenerators>
<ARMA name='Price' variable="Signal">%BASE_WORKING_DIR%/../../../TSA/Sine/arma.pk</ARMA>
<ARMA name='Price' variable="Signal">%HERON_DATA%/TSA/Sine/arma.pk</ARMA>
<Function name="transfers">transfers.py</Function>
</DataGenerators>

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@
</Components>

<DataGenerators>
<ARMA name='signal' variable="WIND">%BASE_WORKING_DIR%/../../../TSA/NYISO/nyiso_arma_2yr.pk</ARMA>
<ARMA name='signal' variable="WIND">%HERON_DATA%/TSA/NYISO/nyiso_arma_2yr.pk</ARMA>
</DataGenerators>

</HERON>
Original file line number Diff line number Diff line change
Expand Up @@ -123,7 +123,7 @@
</Components>

<DataGenerators>
<ARMA name='Price' variable="Signal">%BASE_WORKING_DIR%/../../../TSA/Sine/arma.pk</ARMA>
<ARMA name='Price' variable="Signal">%HERON_DATA%/TSA/Sine/arma.pk</ARMA>
<Function name="transfers">transfers.py</Function>
</DataGenerators>

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,6 @@
</Components>

<DataGenerators>
<CSV name='flex' variable="Signal">%BASE_WORKING_DIR%/../../../mechanics/static_history/Static.csv</CSV>
<CSV name='flex' variable="Signal">%HERON_DATA%/mechanics/static_history/Static.csv</CSV>
</DataGenerators>
</HERON>
Original file line number Diff line number Diff line change
Expand Up @@ -121,7 +121,7 @@
</Components>

<DataGenerators>
<ARMA name='Price' variable="Signal">%BASE_WORKING_DIR%/../../../TSA/Sine/arma.pk</ARMA>
<ARMA name='Price' variable="Signal">%HERON_DATA%/TSA/Sine/arma.pk</ARMA>
<Function name="transfers">transfers.py</Function>
<Function name="storage_control">run_storage.py</Function>
</DataGenerators>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -107,7 +107,7 @@
</Components>

<DataGenerators>
<ARMA name='flex' variable="Signal">%BASE_WORKING_DIR%/../../../TSA/Sine30yr/arma_30yr.pk</ARMA>
<ARMA name='flex' variable="Signal">%HERON_DATA%/TSA/Sine30yr/arma_30yr.pk</ARMA>
</DataGenerators>
</HERON>

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -162,7 +162,7 @@
</Components>

<DataGenerators>
<ARMA name='Price' variable="Signal">%BASE_WORKING_DIR%/../../../TSA/Sine/arma.pk</ARMA>
<ARMA name='Price' variable="Signal">%HERON_DATA%/TSA/Sine/arma.pk</ARMA>
</DataGenerators>

</HERON>
Original file line number Diff line number Diff line change
Expand Up @@ -116,7 +116,7 @@
</Components>

<DataGenerators>
<ARMA name='flex' variable="Signal">%BASE_WORKING_DIR%/../../../TSA/Sine30yr/arma_30yr.pk</ARMA>
<ARMA name='flex' variable="Signal">%HERON_DATA%/TSA/Sine30yr/arma_30yr.pk</ARMA>
</DataGenerators>
</HERON>

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,7 @@
</Components>

<DataGenerators>
<ARMA name='Price' variable="Signal">%BASE_WORKING_DIR%/../../../TSA/Sine/arma.pk</ARMA>
<ARMA name='Price' variable="Signal">%HERON_DATA%/TSA/Sine/arma.pk</ARMA>
<Function name="transfers">transfers.py</Function>
</DataGenerators>

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@
</Components>

<DataGenerators>
<ARMA name='Speed' variable="Signal">%BASE_WORKING_DIR%/../../../TSA/Sine/arma.pk</ARMA>
<ARMA name='Speed' variable="Signal">%HERON_DATA%/TSA/Sine/arma.pk</ARMA>
<Function name="transfer">transfers.py</Function>
</DataGenerators>

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@
</Components>

<DataGenerators>
<ARMA name='Speed' variable="Signal">%BASE_WORKING_DIR%/../../../TSA/Sine/arma.pk</ARMA>
<ARMA name='Speed' variable="Signal">%HERON_DATA%/TSA/Sine/arma.pk</ARMA>
<Function name="transfers">transfers.py</Function>
</DataGenerators>

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -268,7 +268,7 @@
</Components>

<DataGenerators>
<ARMA name='LMP' variable="Signal">%BASE_WORKING_DIR%/../../../../TSA/Sine/arma.pk</ARMA>
<ARMA name='LMP' variable="Signal">%HERON_DATA%/TSA/Sine/arma.pk</ARMA>
</DataGenerators>

</HERON>
Loading
Loading