Got the error: eppy.bunch_subclass.BadEPFieldError: unknown field Program_Line_502

[dsanchez-garcia]

Hi,

I am working on an EMS script. I need to add a very long EMS Program object with eppy. The previous version of the EMS script had roughly 450 lines and it worked, but now I have increased it to roughly 1400. So, when I run the python script to add the EMS Program, I get the error on the subject:

  File "C:\Users\user\PycharmProjects\accim\accim\accim\sim\accim_Base_EMS.py", line 346, in addEMSProgramsBase
    self.idf1.newidfobject(
  File "C:\Users\user\AppData\Local\Programs\Python\Python39\lib\site-packages\eppy\modeleditor.py", line 759, in newidfobject
    abunch[k] = v
  File "C:\Users\user\AppData\Local\Programs\Python\Python39\lib\site-packages\eppy\bunch_subclass.py", line 353, in __setitem__
    raise BadEPFieldError(astr)
eppy.bunch_subclass.BadEPFieldError: unknown field Program_Line_502

Is this related to the number of lines? Can you please amend it somehow?

Thank you very much in advance. Kind regards.

[JasonGlazer]

I believe you can fix this yourself in the Energy+.idd file. If you open that file with a text editor and go to the EnergyManagementSystem:Program you will notice that it has fields listed to A502. If you want to extend that, just keep adding more fields by copying and renumbering. Make sure that intermediate fields are followed by a comma and the last field is followed by a semicolon.

[dsanchez-garcia]

I believe you can fix this yourself in the Energy+.idd file. If you open that file with a text editor and go to the EnergyManagementSystem:Program you will notice that it has fields listed to A502. If you want to extend that, just keep adding more fields by copying and renumbering. Make sure that intermediate fields are followed by a comma and the last field is followed by a semicolon.

You are correct Jason. I have amended it manually and it worked. Thank you very much. Do you know if there's a way to amend it by means of eppy? Thinking of a script to automate the amendment.

[JasonGlazer]

I don't think anything that automates that has been developed but it would be handy.

[santoshphilip]

@JasonGlazer , I agree that It would be good to automate this.
I had taken a look at it earlier and the code here is slightly tricky. I'll take another stab at doing it.

@dsanchez-garcia , You may need to do this manually for a while. This will take a little time to do this update. Hopefully it can be done in such a way that IDD update automatically behind the scenes.

[dsanchez-garcia]

@santoshphilip @JasonGlazer thank you both. I did a little piece of code to modify the idd as a text file that will work temporarily. Could you please let me know when you do it? Thanks!

[santoshphilip]

Keep an eye on this issue (turn on notification). I'll close this issue with a message once it is fixed.

Can you share the code you used to modify the IDD?

[dsanchez-garcia]

Of course. It only works for versions 9.0.0-1-2-3-4-5-6. I was struggling with the style at the comment, so I have attached a zip file with the python file. Hope it helps.

amend_IDF.zip

[santoshphilip]

thanx

[santoshphilip]

I think I am pretty close to solution for this. (*** it was really hard ****)
Couple of new questions have popped up while working on the issue.

I'll do an initial release and then start to open additional issues after a bit of discussion here

[santoshphilip]

I have a limited fix for this.

:Problem: If the IDF files has more extensible fields than the IDD file for an idfobject, the read will fail
:Solution: Increase the extensible fields of the IDD (in memory) to match the IDF file. It will not change the Energy+.idd on the disk.

• This fix works if you are reading a file that has more extensible fields than are present in the IDD
• this will happen transparently and automatically - no additional documentation is needed
• eppy will still break in the following circumstances:
  • idf.newidfobject() -> if you make a new object with more fields than is in the IDD
  • idfobject.fieldname = somevalue -> where the fieldname goes beyond the extensible field in the IDD
  • idfobject['fieldname'] = somevalue -> where the 'fieldname' goes beyond the extensible field in the IDD
  • QUESTION: are there other cases where failure will occur - that I have not thought of

Other options:

• Maybe write a script that will increase the fields in the IDD file on disk ??
• Maybe write an function that the user can call that will increase fields of the IDD in memory ??
• I don't like the above two options because it forces the user to explicitly deal with the IDD
• Best would be if the user does not have to think about the IDD at all

I have merged the fix into the develop branch - not made a release yet.

I'll open new issues for the problems mentioned above. Please post any comments. I'll close this issue after your comments @JasonGlazer and @dsanchez-garcia

[dsanchez-garcia]

The first one of the other options is what I did as a temporary solution. I wrote a very simplistic script as a PostInstallCommand to automatically increase the fields of EnergyManagementSystem:Program in the IDD and save a backup file after the package is installed, so that the user don't need to deal with the IDD. I increased the fields up to 7500, so it is possible but unlikely that the user has a program longer than this. Maybe doing this for the objects that might need it is a solution.

[JasonGlazer]

I agree that something automatic would be best.

I guess if you try to assign to a name that does not exist in the IDD, you could see if the "root" name of the field without any trailing number matching the last field(s) in the IDD without a number and if it does, just increase the size of the input object in the IDD for that input object to that number. That would help with the second and third issue you pointed out.

As far as on disk versus in memory, it seems likely that the user will be manipulating files that need this larger input objects on a repeated basis so maybe changing the IDD on the disk is the way to go. Another option is that you could have a "size.ini" file that is written and read by eppy that would list any input objects that need to be increased in size and what size to use. It could be read each time eppy is started and written when ever this resizing needs to take place.

[santoshphilip]

@JasonGlazer , All good ideas.
I'll put my thinking hat on this and consider how to implement (and what to implement)

[santoshphilip]

I think this issue has been fixed.
The IDD is transparently extended in memory when needed. This happens when

1. eppy reads a file that has fields that extend beyond what is in the IDD
2. when newidfobject( ... ) is called with fields that extend beyond what is in the IDD
3. when idfobject.fieldname_n = xxx when extendable field fieldname_n is not in the IDD
4. when idfobject["fieldname_n"] = xxx when extendable field fieldname_n is not in the IDD
5. value = idfobject.fieldname_n when extendable field fieldname_n is not in the IDD
   • value = ""
6. value = idfobject["fieldname_n"] when extendable field fieldname_n is not in the IDD
   • value = ""

@JasonGlazer , Can you think of any other possibilities where the IDD needs to be extended
@dsanchez-garcia , can you use the code in the develop branch and try to break it? Everything should work without making any changes to the IDD on disk.

[santoshphilip]

In fixing this issue, I had to make changes in parts of the code that are fragile (I wrote some of it long ago, when I was not good at it.)

So I am a little concerned that I may have broken things.

So @dsanchez-garcia , I would be happy if you are able to show that it is broken. Try and break it. Because, then it can be fixed.

If you are not sure how to switch to the develop branch, ask me.

[dsanchez-garcia]

No problem, I'll try to break it. But I am not sure how to switch to the develop branch, so if you could provide some guidance, I would appreciate it.

[santoshphilip]

if you are using virtualenv and pip here is how you can do it.
Make a virtual env for the develop branch

python3 -m venv ~/venvs/eppy_develop 
source ~/venvs/eppy_develop/bin/activate 

pip install the develop branch

pip install https://github.com/santoshphilip/eppy/archive/develop.zip

You can test run this code. (update pathname for Energy+.idd)

"""test out develop branch. 
Opens a file with 5000 extensible variables 
when the IDD has only 48 extensible variables"""

import eppy
from io import StringIO
from eppy.modeleditor import IDF

astr = """
Version,
    9.6;                      !- Version Identifier

WINDOWMATERIAL:GLAZINGGROUP:THERMOCHROMIC,
    Gumby,                    !- Name
    55,
    G1,"""

nn = 5000
# nn = 50
extfields = ",".join([f"{i}, G{i}" for i in range(nn)])
newstr = f"{astr} {extfields};"

fhandle = StringIO(newstr)
IDF.setiddname("/Applications/EnergyPlus-9-6-0/Energy+.idd")
idf = IDF(fhandle)
wm = idf.idfobjects["WINDOWMATERIAL:GLAZINGGROUP:THERMOCHROMIC"]
assert wm[0][f"Optical_Data_Temperature_{nn + 1}"] == nn - 1
assert wm[0][f"Window_Material_Glazing_Name_{nn + 1}"] == f"G{nn - 1}"


# or to be more explicit
assert wm[0].Optical_Data_Temperature_5001 == 4999
assert wm[0].Window_Material_Glazing_Name_5001 == "G4999"

I have tested all the above steps on my computer

[santoshphilip]

here is how I tested the changes

https://github.com/santoshphilip/eppy/blob/develop/tests/test_extend_extensible.py

[dsanchez-garcia]

thanks @santoshphilip ; apologies for the delay, i've been busy lately. I'll do it as soon as I can.

[santoshphilip]

Not a problem. Take your time over it.

If you can break it - great !
If it doesn't break, we are going to trust that it is good :-)

[dsanchez-garcia]

@santoshphilip just tested both and passed:

C:\Python\pythonProject\venv\Scripts\python.exe "C:\Program Files\JetBrains\PyCharm Community Edition 2020.2.3\plugins\python-ce\helpers\pycharm_jb_pytest_runner.py" --path C:/Python/pythonProject/tests/test_2.py
Testing started at 9:39 ...
Launching pytest with arguments C:/Python/pythonProject/tests/test_2.py --no-header --no-summary -q in C:\Python\pythonProject\tests

============================= test session starts =============================
collecting ... collected 3 items

test_2.py::test_read_overextended PASSED [ 33%]
test_2.py::test_newidfobject_overextend PASSED [ 66%]
test_2.py::test_getset_overextended PASSED [100%]

============================== 3 passed in 1.17s ==============================

Process finished with exit code 0

Thanks, and apologies for the delay.

[santoshphilip]

I am going to say that is is good for release.
I am travelling for the next week.
I'll do a release once I get back.

[santoshphilip]

All these changes have been published in release r0.5.60
Now IDD files do not have to be extended by hand anymore