running osw when measure not found, instead of stopping it keeps running the osw and E+.

[DavidGoldwasser]

Issue overview

Issue with how workflows where one or more measures can't be found. It skips over those measure and runs the measure it can find. In cases where some measure are found and others are not it really makes mess out.osw putting results from one measure over the wrong position.

Current Behavior

Test Case A (running bar_typical from osw2osa repo): https://github.com/NREL/osw2osa/blob/master/workflows/bar_typical/in.osw

• The workflow fails with no measure running. E+ runs with an error. Below is relevant parts of out.osw. Note current step shows 0. When all measure are run at end current step shows as 7. None of the measure in out.osw have any results.
  Relevant elements of out.osw
  "completed_at" : "20240806T065546Z",
  "completed_status" : "Fail",
  "current_step" : 0,
  "eplusout_err" : ". . . . . . ** Severe ** GetNextEnvironment: Weather Environment(s) requested, but no weather file found\n ** Fatal ** Due to previous error condition, simulation terminated\n . . . . . ",
  "started_at" : "20240806T065545Z",

Test Case B (running floorspace_typical from osw2osa repo) https://github.com/NREL/osw2osa/blob/master/workflows/floorspace_typical/in.osw

• Similar to Test case a no measures are found but and E+ runs, but because in this case the seed model had enough to run the E+ simulation was successful and even produced an eplusout.html, although no reporting measure were run or produced, but I think only because the measures were not found. When all measure are run current step would show as 7.
  Relevant elements of out.osw
  "completed_at" : "20240806T065546Z",
  "completed_status" : "Success",
  "current_step" : 0,
  "eplusout_err" : " . . . . ******* EnergyPlus Sizing Error Summary. During Sizing: 2 Warning; 0 Severe Errors.\n ************* EnergyPlus Completed Successfully-- 85 Warning; 0 Severe Errors; Elapsed Time=00hr 00min 0.32sec\n"
  "started_at" : "20240806T065546Z",

Test Case C (running ze_retail from osw2osa repo) https://github.com/NREL/osw2osa/blob/master/workflows/ze_retail/in.osw

• Ok, os here is where things get weird. There are 16 measure in this repo, and only 2 of them would be found although the workflow failed and stoped on the first of two measures. This results in an odd out.osw where results are shown for the 11th measure but on top of the first measure (that wasn't found. I'll show in.osw relevant parts of in.osw here first for clarity.

Relevant elements of in.osw for test case C
{
"file_paths" :
[
"../../../weather",
"../../../seeds",
"../../../files"
],
"hash" : "4EDC1148",
"measure_paths" :
[
"../../../measures"
],
"seed_file" : "aedg_retail.osm",
"steps" :
[
{
"arguments" :
{
"set_year" : "2019",
"weather_file_name" : "USA_GA_Atlanta-Hartsfield-Jackson.Intl.AP.722190_TMY3.epw"
},
"measure_dir_name" : "ChangeBuildingLocation"
},
{
"arguments" :
{
"SKIP" : false,
"facade" : "North",
"wwr" : "0.0"
},
"measure_dir_name" : "SetWindowToWallRatioByFacade"
}, (14 more measures follow this)

Relevant elements of out.osw for test case C. Note that results for the 11th measure ze_retail_typical are written on top of the first measure ChangeBuildingLocation. This was the most confusing of the test cases because it looked like ChangeBuildingLocation ran, but it really didn't. When measure are found current step shows at 11.

{
"completed_at" : "20240806T065547Z",
"completed_status" : "Fail",
"current_step" : 1,
"file_paths" :
[
"/Users/dgoldwas/Documents/github/nrel/osw2osa/run/workflows/ze_retail/generated_files",
"../../../weather",
"../../../seeds",
"../../../files",
"./files",
"./weather",
"../../files",
"../../weather",
"./"
],
"hash" : "8CD3DD8F",
"measure_paths" :
[
"../../../measures"
],
"run_options" : null,
"seed_file" : "aedg_retail.osm",
"started_at" : "20240806T065545Z",
"steps" :
[
{
"arguments" :
{
"set_year" : "2019",
"weather_file_name" : "USA_GA_Atlanta-Hartsfield-Jackson.Intl.AP.722190_TMY3.epw"
},
"measure_dir_name" : "ChangeBuildingLocation",
"result" :
{
"completed_at" : "20240806T065547Z",
"measure_class_name" : "ZeRetailCreateTypicalBuildingFromModel",
"measure_display_name" : "ZE Retail Create Typical Building from Model",
"measure_name" : "ze_retail_create_typical_building_from_model",
"measure_taxonomy" : "Whole Building.Space Types",
"measure_type" : "ModelMeasure",
"measure_uid" : "05fcdf1f-8266-4428-9ba2-da9fa94dceec",
"measure_version_id" : "07aa88f1-ec5d-4442-8d50-db3c3228abb8",
"measure_version_modified" : "20201118T053018Z",
"measure_xml_checksum" : "EE13F09F",
"started_at" : "20240806T065547Z",
"stderr" : "",
"stdout" : "[openstudio.model.YearDescription] 'UseWeatherFile' is selected in YearDescription, . . . . "
"step_errors" :
[
"SWIG director method error. NoMethodError: undefined metho model_get_building_climate_zone_and_building_type for #<NRELZNEReady2017:0x000000012abd6070 @template="NREL ZNE Ready 2017", @standards_data={"hvac_inference"=>[{"template"=>"NREL ZNE Ready 2017", "size_categor . . . . . ."
"step_files" : [],
"step_info" :
[
"Adding loads to space type named Office_Restroom - NREL ZNE Ready 2017",
"Adding loads to space type named Conference - NREL ZNE Ready 2017",
"Adding loads to space type named Break_Room - NREL ZNE Ready 2017",
"Adding loads to space type named Back_Space - NREL ZNE Ready 2017",
"Adding loads to space type named Retail - NREL ZNE Ready 2017",
"Adding loads to space type named Point_of_Sale - NREL ZNE Ready 2017",
"Adding loads to space type named Front_Retail - NREL ZNE Ready 2017",
"Adding loads to space type named Closed_Office - NREL ZNE Ready 2017",
"Adding loads to space type named Entry - NREL ZNE Ready 2017",
"Adding loads to space type named Mechanical_Room - NREL ZNE Ready 2017",
"Adding loads to space type named Core_Retail - NREL ZNE Ready 2017"
],
"step_initial_condition" : "The building started with 220 objects.",
"step_result" : "Fail",
"step_values" :
[
{
"name" : "template",
"value" : "NREL ZNE Ready 2017"
},
{
"name" : "template_space_loads",
"value" : "ZE AEDG Multifamily"
},
{
"name" : "system_type",
"value" : "PSZ-HP"
},
{
"name" : "hvac_delivery_type",
"value" : "Forced Air"
},
{
"name" : "htg_src",
"value" : "NaturalGas"
},
{
"name" : "clg_src",
"value" : "Electricity"
},
{
"name" : "swh_src",
"value" : "Electricity"
},
{
"name" : "kitchen_makeup",
"value" : "Adjacent"
},
{
"name" : "exterior_lighting_zone",
"value" : "3 - All Other Areas"
},
{
"name" : "add_constructions",
"value" : true
},
{
"name" : "add_space_type_loads",
"value" : true
},
{
"name" : "add_elevators",
"value" : true
},
{
"name" : "add_internal_mass",
"value" : true
},
{
"name" : "add_exterior_lights",
"value" : true
},
{
"name" : "onsite_parking_fraction",
"value" : 1.0
},
{
"name" : "add_exhaust",
"value" : true
},
{
"name" : "add_swh",
"value" : true
},
{
"name" : "add_thermostat",
"value" : true
},
{
"name" : "add_hvac",
"value" : true
},
{
"name" : "add_refrigeration",
"value" : false
},
{
"name" : "modify_wkdy_op_hrs",
"value" : false
},
{
"name" : "wkdy_op_hrs_start_time",
"value" : 8.0
},
{
"name" : "wkdy_op_hrs_duration",
"value" : 8.0
},
{
"name" : "modify_wknd_op_hrs",
"value" : false
},
{
"name" : "wknd_op_hrs_start_time",
"value" : 8.0
},
{
"name" : "wknd_op_hrs_duration",
"value" : 8.0
},
{
"name" : "unmet_hours_tolerance",
"value" : 1.0
},
{
"name" : "remove_objects",
"value" : false
},
{
"name" : "use_upstream_args",
"value" : false
},
{
"name" : "enable_dst",
"value" : true
},
{
"name" : "climate_zone",
"value" : "Lookup From Model"
}
],
"step_warnings" : []
}
},
{
"arguments" :
{
"SKIP" : false,
"facade" : "North",
"wwr" : "0.0"
},
"measure_dir_name" : "SetWindowToWallRatioByFacade"
},

Still for Test Case C here is part of the out.osw if all measure were found. Note that the results for ChangeBuildingLocation are actually from ChangeBuildingLocation.

{
"completed_at" : "20240806T065132Z",
"completed_status" : "Fail",
"current_step" : 11,
"file_paths" :
[
"/Users/dgoldwas/Documents/github/nrel/osw2osa/run/workflows/ze_retail/generated_files",
"../../../weather",
"../../../seeds",
"../../../files",
"./files",
"./weather",
"../../files",
"../../weather",
"./"
],
"hash" : "CE6AE42F",
"measure_paths" :
[
"../../../.bundle/install/ruby/3.2.0/gems/openstudio-calibration-0.10.0/lib/measures",
"../../../.bundle/install/ruby/3.2.0/gems/openstudio-common-measures-0.10.0/lib/measures",
"../../../.bundle/install/ruby/3.2.0/gems/openstudio-ee-0.10.0/lib/measures",
"../../../.bundle/install/ruby/3.2.0/gems/openstudio-model-articulation-0.10.0/lib/measures",
"../../../.bundle/install/ruby/3.2.0/gems/tbd-3.4.2/lib/measures",
"../../../measures"
],
"run_options" : null,
"seed_file" : "aedg_retail.osm",
"started_at" : "20240806T065043Z",
"steps" :
[
{
"arguments" :
{
"set_year" : "2019",
"weather_file_name" : "USA_IL_Chicago-OHare.Intl.AP.725300_TMY3.epw"
},
"measure_dir_name" : "ChangeBuildingLocation",
"result" :
{
"completed_at" : "20240806T065045Z",
"measure_class_name" : "ChangeBuildingLocation",
"measure_display_name" : "ChangeBuildingLocation",
"measure_name" : "change_building_location",
"measure_taxonomy" : "Whole Building.Space Types",
"measure_type" : "ModelMeasure",
"measure_uid" : "d4db4971-f5ba-11e3-a3ac-0800200c9a66",
"measure_version_id" : "746afe91-d190-49c5-9534-26416d0bc55a",
"measure_version_modified" : "20240711T143210Z",
"measure_xml_checksum" : "057E8D9D",
"started_at" : "20240806T065045Z",
"stderr" : "",
"stdout" : "[openstudio.model.YearDescription] 'UseWeatherFile' is selected in YearDescription, but there are no weather file set for the model.\n[openstudio.model.YearDescription] 'UseWeatherFile' is selected in YearDescription, but there are no weather file set for the model.\n",
"step_errors" : [],
"step_files" : [],
"step_final_condition" : "The final weather file is Chicago Ohare Intl Ap and the model has 3 design day objects.",
"step_info" :
[
"city is Chicago Ohare Intl Ap. State is IL",
"Changing Calendar Year to 2019,",
"mean dry bulb is 9.924999999999999",
"Adding object Chicago Ohare Intl Ap Ann Clg .4% Condns DB=>MWB",
"Adding object Chicago Ohare Intl Ap Ann Clg .4% Condns WB=>MDB",
"Adding object Chicago Ohare Intl Ap Ann Htg 99.6% Condns DB",
"Setting ASHRAE Climate Zone to 5A"
],
"step_initial_condition" : "No weather file is set. The model has 0 design day objects",
"step_result" : "Success",
"step_values" :
[
{
"name" : "weather_file_name",
"value" : "USA_IL_Chicago-OHare.Intl.AP.725300_TMY3.epw"
},
{
"name" : "climate_zone",
"value" : "Lookup From Stat File"
},
{
"name" : "set_year",
"value" : 2019
},
{
"name" : "use_upstream_args",
"value" : true
},
{
"name" : "epw_gsub",
"value" : "Do Nothing"
},
{
"name" : "reported_climate_zone",
"value" : "5A"
}
],
"step_warnings" : []
}
},

Expected Behavior

In the past when a measure in a workflow could not be found the workflow would fail and there would be a warning that a measure could not be found or loaded. This info didn't make it into out.osw. I believe the workflow would fail to start to run even if missing measure was not first measure in the workflow.

I have not confirmed but this may have changed from Ruby based Workflow gem switch to C++ workflow. I more minor changes to investigate is workflows with out of order measures (where they are not model, E+, then reporting measure). This used to also fail but now will be more flexible which would have it run measure out of order if a model measure is listed after an E+ measure, or an E+ measure after a reporting measure.

Steps to Reproduce

1. Setup a workflow with one or more measures that can't be found in measure paths
2. Run workflow
3. Look at out.osw and resulting files

Possible Solution

Check for missing measures either up front or as each measure is run and have it halt the workflow. An enhancement over prior behavior woudl be to make message for measure that can't be found or measure that can't be loaded into the out OSW at the proper step of the workflow.

Details

Environment

Tested OpenStudio 3.8.0 Mac

Context

This isn't a blocker but is problematic because it's possible that a workflow may appear to run, even to show as being successful but one or more measures are skipped without user being made aware of this.

[joseph-robertson]

A related issue is trying to run an OSW where measures are out of order. For example:

• ModelMeasureA
• ModelMeasureB
• ReportingMeasureA
• ModelMeasureC
• ReportingMeasureB

This should stop the workflow and finish as failed.

[jasondegraw]

I have seen this on Ubuntu with OpenStudio 3.7, I just didn't realize what was happening. My "measure_paths" value was messed up so none of the measures were being found, but I had a configuration issue that masked what was happening. When fixing the configuration, I must have fixed my measure path. This came up recently when I went back to rerun things that had the original wrong measure path in the OSW but I set it up properly this time. The workflow ran to completion without errors but none of the measures were applied and the results were very much not what was expected.

[DavidGoldwasser]

In case it isn't documented above this also happens on OSW where a measure is a measure with "SKIP": true,. This made issues for workflows.

The specific use case of skipped measure was fixed sometime between 3.7 and 3.8. It still happens in 3.7. It doesn't happen in 3.8 and 3.9. I still believe this issue for missing measure paths is still valid. I can make a zip file replicating this after 3.9 is out and will target this for 3.10 instead. It is more of an edge case that would be with mall-formed OSW files or resource paths.

[DavidGoldwasser]

@jmarrec I'll re assign you after I can setup a good example.

[jmarrec]

OpenStudio/src/workflow/RunInitialization.cpp

Lines 59 to 60 in cc1e0bb

	// TODO: Validate the OSW measures if the flag is set to true, (the default state)
	// Note JM 2022-11-07: Is it better to try and load all measures once, instead of crashing later?

This corresponds to:
https://github.com/NREL/OpenStudio-workflow-gem/blob/39c84e45f152446a92379a501807e2691dba15ab/lib/openstudio/workflow/jobs/run_initialization.rb#L89-L94

and the validate_measures is here:
https://github.com/NREL/OpenStudio-workflow-gem/blob/39c84e45f152446a92379a501807e2691dba15ab/lib/openstudio/workflow/util/measure.rb#L97-L151