New Crop Livestock Enterprise Model (CLEM)

[lie112]

CLEM is a whole-of-farm enterprise model that adopts the functionality of precursor models (Integrated Analysis Tool for smallholders, IAT and North Australia Beef Systems Analyser, NABSA) to provide a model that can test a range of farm improvement strategies in a multitude of crop and livestock enterprises while tracking impacts on finances, natural resources, and highly constrained resources such as labour. To achieve this, CLEM expands upon the approach at the core of IAT where available Resources or farm assets (e.g. animals, crops, fodder, labour, land, money, water) are used by farm Activities (e.g. grow crops, manage animals, feed animals, manage fodder, manage manure, feed household). The APSIM NG environment with graphical user interface and CLEM-specific additions provides a framework for the setup, parameterisation and control of this type of model. This will also allow for dynamic linkage between the whole farm model and APSIM crop models.

[sarahcleary]

@lie112 - is this ready for RP (final) review?

[lie112]

This version is ready for review and includes the requested example files and validation tests, Of course there are lots of things I am still working on, but this is ready to review.

[sarahcleary]

@APSIMInitiative/reference-panel - This is ready for your final review. Please note the updates including the example files and validation tests.
To review - please download the latest version of APSIMX.
Email correspondence from Di Prestwidge:
Dean,
Adam and I have pulled together a set of Example and Sensibility simulations and associated data files for the APSIM Ref Panel to finalise the CLEM submission.
The CLEM Example Simulations demonstrate the functionality of CLEM and the 2 simple Sensibility simulations are evidence that CLEM works, documented with simple hypotheses and graphs. (There is a larger suite of Sensibility simulations that we have kept aside to share with stakeholders)
We have also attached the current version of the CLEM HELP documentation (Electronic version and a PDF version). The Electronic version is accessible (linked) from within CLEM. We ask if it could be hosted with the APSIM documentation so that it is easily accessible. We will be updating this regularly.
Please let us if there is anything else you require.
Regards
Di and Adam

CLEM_Example_Simulations:

• Cropping Examples (CLEM_Example_Cropping.apsimx ) - Single and Mixed Cropping and Rotation Cropping
  o With data files FileCrop_Example.prn, FileVegetables_Example2.prn, FileCrop_SorgWheat3.prn
• Grazing Example (CLEM_Example_Grazing.apsimx)
  o With data file - FilePasture_ExampleDB.db (reduced to years 2000-2010)
  CLEM Sensibility test files:
• CLEM_Sensibility_GrowCrop.apsimx
• CLEM_Sensibility_IndividualAnimal.apsimx

CLEM HELP documentation:

• \Electronic – open “Default.htm”
• \PDF – “A4Electronic.pdf”

CLEM_Example_Simulations.zip
CLEM_Sensibility_tests.zip
CLEMHelp20181129.zip

Any questions, please post here.

[sarahcleary]

@pre078 - CLEM Issue for your reference

[Keith-Pembleton]

Running the example grazing simulation and i get the following warning
"No conception details were found for [r=Bos indicus]
Please add a conception component below the [r=RuminantType]"

Also the graphs of this simulation don't appear to be displaying properly but i think this is because the simulation is only running for a year rather than the 10 years that the clock is set too?
Cheers
Keith

[Keith-Pembleton]

There appears to be a file missing that is preventing me from running the crop growth sensibility simulation. it appears "Crop_new_format(Burkina 20130222).prn" is needed but is not in the zip files provided
Cheers
Keith

[zur003]

The CLEM grazing example works fine for me.

[hol430]

Ah - it works fine for me as well from my compiled version. Throws an error when run from the installed version though, probably because the .prn file is being ignored by InnoSetup when we create the installer.

[Keith-Pembleton]

Whats the solution Drew?

[zur003]

@Keith-Pembleton - try upgrading to the most current release. I was able to replicate your "no conception details" problem with an older release, but the current one seems OK.

[hol430]

I just had a look and the .prn file is actually included with the installed version, so that doesn't seem to be the problem. What's more, I've realised I copied the .apsimx file without copying the .prn file, which is why I was getting a failure.

The upshot is that I'm not sure why it's not working for you. Did you open the .apsimx file by clicking 'open an example'? If so, then you would have saved the file somewhere else on your computer. You may need to also copy the .prn file into the same folder as the .apsimx file.

Also, some extra files which Di wanted posted on this thread:

CropFilesBurkina.zip

[Keith-Pembleton]

@zur003 Eric I was running version 201.3.29.3669 (updated just before I started looking at it) and have just updated to 2019.4.9.3719, but still get the no conception details problem. Thoughts anyone?

[Keith-Pembleton]

@hol430 Thanks Drew, the files you provided have allowed me to get the crop management crop growth sensibility simulation working.

[hol430]

Yeah the CLEM grazing example still fails for me if I save it into a different folder:

@error:Could not locate file [o=C:\­Users\­hol430\­Downloads\­FileForage.prn] for [x=FileForage]

It's the same problem as #3271 and #1783 - Apsim will get the path wrong after doing the following steps:

1. Click "Open an Example" and open one of the examples (e.g. factorial example)
2. Save as -> save into a different folder (e.g. desktop)
3. Run

For the user, this means that you have to go to the weather node (or the file forage node for the CLEM example), and update the path to the file.

[sarahcleary]

@APSIMInitiative/reference-panel - does anyone have any further comments?

[jbrider]

A message box appears showing version number when browsing the tree - when clicking the Clem component. Is it supposed to be appearing in the messages tab?

[peter-devoil]

If the examples Crop File names are changed to use (eg) "%root%/Examples/CLEM/FileVegetables_Example2.prn", the simulation will work out of the box.

In response to keith's request to "test clem":

Overall - it's not clear whether the "product" is clem, or apsimx. A user runs the apsimx program, opens apsimx files, but the simulation contents have clem.zoneclem etc. In the lack of a plan to merge clem and apsim components, perhaps apsim & clem should be differentiated: (eg add a new file type .clem that is internally the same format)

• UI Summary tab (eg Simulation/CLEM/Resources/Land/Padd1): Not clear why parameters (in the blue boxes) aren't editable. If they're not editable they should be in plain, unadorned text.

• • Why not combine the Summary and Properties tabs? It's the same info.
• • What's the point of a Versions tab if the version info is also in the header? Shouldn't a summary of functionality be in the summary tab - and not deduced from the version history?

• There's no way to display the data in a fileCrop?

• Units should be mandatory, on every entry. Is land area in ha, or m^2?

• Land/Field/Soil type index. This is 2019. Surely we can have more descriptive keys than just numbers? Why aren't they list items?

[LouisAK]

I tried the example files and they worked fine for me.
Agree with Peter... it will be good if the data in File Crop can be displayed.

[sarahcleary]

@pre078 - please note discussion above.

[pre078]

@LouisAK - With reference to FileCrop display - we had turned this feature off as large datasets were slowing it down (ie. GRASP db files). We are re-thinking several options for linking in external data. Will update soon.

[pre078]

@peter-devoil - Thanks for your comments - will wait for Adam Liedloff to reply to most of them.
Re: Units - this is on our radar - part of our to-do list
Re: FileCrop display - switched this off for large datasets - currently re-thinking options for external data
Re: Land/Field/Soil type index - this is a legacy from IAT input just to get it working - to be updated as part of re-thinking FileCrop options

[lie112]

Hi everyone,
I will attempt to answer the comments above.
Missing files
Sorry that there were issues with missing files for some people. If all files including the APSIMx file are in the same folder each simulation will run successfully. If there are still missing files let me know.
Display data file contents
We have removed the data file data display due to some large files (9-105 MB) not loading quickly. We figured these text or database data files were known by the user and could be viewed in other software packages.
Missing conception details
This is a updated feature of the latest version of CLEM where conception details were moved to a component to allow a range of styles to be used by the user including a new condition based individual conception rate. The current release method means that the version available to users is the latest version of code submitted and unfortunately the APSIMx file was generated based on the version thought to remain for testing phase. A new APSIMx file for grazing with conception details is available from Di or myself, but should have been updated with the new code.
I am thinking through how APSIM can resolve this issue as any update of the model can result in loss of backward compatibility. This is another reason why I added the versioning to point the user towards a fix. The messages tab of the main CLEM component will also display a cleaned up, HTML formatted error message and often provides a solution. Adding a conception component below the RuminantType resource as stated would have fixed the problem but of course you would not have known what values to supply in the testing setting without an informed user.
Soil type
As noted by Di this is a legacy index from IAT and NABSA and was maintained to keep old data set compatibility. The upcoming version of CLEM allows a string value to be supplied to identify the soil type/land type as we do find this a valuable means of selecting a subset of data from input files and the soil type could be taken from the simulation name column of APSIMx crop output files. There is no means of making this a drop down list as we have no way of determining the full list of options available to the user. We have endeavoured to add a lot of additional user input beyond what was available in APSIM to help our users such as fully propagated drop down lists to avoid typo issues and restrict options available.
Editing values in summary display
The summary tab is designed as a added feature of allow the user to get a descriptive outline of the settings they have applied. Reading these plain English pages should highlight things that may not have been set correctly and show where there are problems before running the simulation. This is important in models containing lots of parameters and switches and building complex component based simulation trees where the order of components influences the model outcomes. The summary display generates a HTML page that can be printed or saved. There is currently no means of making the boxed values (used to show values set by the user) in the summary display editable using the GTK# graphic engine of APSIMx. The user should select the Properties tab to access and modify all properties of the selected model component and utilise the categorised groupings of some property pages. CLEM will remember which tab was last selected and return to that tab when the component is again selected. Colour was added to the summary page to identify entered values and keep the resource/activity colour scheme of CLEM.
A full description of each model component is available by clicking the help button in the header which takes you to comprehensive online documentation and directly to the selected component details. This can also be installed on the machine for local help where internet is not available in remote areas.
CLEM as part of APSIMx
The aim from the beginning of developing CLEM was for it to be included in the APSIMx framework and able to communicate with other APSIMx models such as crops and forage. As would be appreciated this will take some collaborative development to have the models interact, Based on this approach we have made CLEM an APSIMx Zone and CLEM can be added to any APSIMx simulation. CLEM must be tightly coupled with APSIMx development to keep working and so is not a stand alone model.
Units
I am currently working out how to address units in CLEM. The customisable nature of CLEM means that many resources don't have a fixed unit, but it is set by the user and different resources in a group may actually have different units. Others do have units such as Land which also contains a unitsToHectarea converter. We need CLEM to run in international settings where the user is not using m^2 or ha . Yet other resources have fixed units such as days for labour availability. This will be improved in future versions.
Version tab
Like APSIM, changes happen in models between releases. I decided to provide a means for APISM to track the changes in models between releases rather than rely on the overall APSIM release number and a large list of changes the user needs to access. The modular nature of CLEM means that some users will not be interested in whole sections of the model capability and the associated changes in that part of the model. Versioning was included so that people discussing the model know they are using the same component, especially when manager script based components can be later added to anyone's simulation tree. The version details is kept with the code base in the cs file as attribute information.
I had considered removing the version number form the header details, but hadn't at this stage.
Wayward message box
There is a situation where the version tab is occasionally loaded as a separate window by GTK#. I have not currently been able to debug this issue, but think it might be a threading issue. I thought it may have only been happening in the development environment. This is an error and can be closed. This may also be related to the time it initially takes to open a large property presenter that could be performed in a background thread by APSIMx.
Please contact me if you have any further questions about CLEM
Adam

[sarahcleary]

@APSIMInitiative/reference-panel - Final decision to be made at July RP meeting. If you have any further issues, comments or questions, please raise them prior to this meeting. Thanks

[sno036]

I looked at the two examples and they ran 'out of the box' (build 3878) so it is good to know that the file location issue has been fixed. I assume there is some 'proper' documentation for CLEM? This should be uploaded to the website and linked to under the category "Science Documentation" - as is there is not enough information for users. Then add a short blurb about the intent, approach, and basic assumptions (probably copied from the real documentation) and the link to the full documentation into the two examples. With that I think ready to go to release. (Note that I didn't try to create something from scratch - I think I would need that full doco to do that.)

[lie112]

CLEM has full online documentation located at http://www.apsim.info/CLEM/Content/Home.htm
This includes a model overview, user guide (how to) and a full description of every CLEM component (Model) as well as in depth discussions on modelling various farm systems and what components to use. The documentation can be access by clicking the help button on any CLEM model page in the user interface and you will be taken to the appropriate section. It follows the CLEM colour scheme. There is also future functionality to install a local copy of the help documentation for people without internet access. This documentation is extensive and still under development. We aim to have full citations throughout this documentation and alot more examples and screen shots as well as a range of how to videos and tutorials in future. The documentation will eventually be available in PDF booklet format. Someone can add additional links to the CLEM document from other relevant places on the APSIM website.

[sno036]

So:

• @hol430 - can you add that URL (http://www.apsim.info/CLEM/Content/Home.htm) under Science Documentation in https://apsimnextgeneration.netlify.com/modeldocumentation/?
• @lie112 - can you add an edited version of your response above (including the URL to the full documentation) as a memo in the two examples?

[hol430]

Done

[lie112]

Information about help in CLEM is now in each example's information memo and will be updated in an upcoming pull from APSIMx. I don't know the syntax to include a clickable link in the memo.

[hol430]

Haven't tested this, but pretty sure you can do this:

[description](url)

e.g.

[Documentation](https://www.apsim.info/CLEM/Content/Home.htm)

[lie112]

Some CLEM users new to APSIMx are struggling to get examples working because they are opening them from the installed Examples/CLEM directory in C:\Program Files\APSIM2019.06.183947\Examples\CLEM with limited permissions. This causes the following issue
"However initially when I tried to open the CLEM examples, the same error message “the data file is not found” popped up again, then the “update” icon did not work, either. "
I am unable to identify this issue from the development environment. I don't know what the exact error is as I can't find that error reported anywhere in APSIM. This could also be related to the problem with missing datafiles reported previously in this issue.
I have informed them to open using the "open example" which I assume asks them to save the file to a local directory, and then move the associated files.
Is there something we should trap here to stop simulations running from program file directories as this also leads to issues trying to save the simulation when Run is selected? Do you allow simulations to run from the Examples directory? Maybe this is just non-trained users.

[lie112]

Help URL and details now added to example memos.

[sarahcleary]

@sno036 - can you confirm this answers your concerns as raised above?

[hol430]

I believe we usually we don't want users modifying the examples which are shipped with apsim. To run them, users should open them, save them to a new location, and then run them.

[lie112]

I suggest APSIM doesn't allow users to run an example file form the examples directory. This would force new users to move them elsewhere rather than assume you can run them from where they are provided. A copy to folder function would be useful that copies the .apsimx file and all associated data files in the example directory.

[sno036]

That is already how it works with examples. Yes I am happy for it to move to release. As with all the models, the doco will be under continuous improvement but this is fine to begin with.

[sarahcleary]

This is now in release