Replies: 11 comments 17 replies
-
|
@araman2, is this using the newest test version of E3SM-Unified (1.8.0rc1) and the previous release (1.7.1)? |
Beta Was this translation helpful? Give feedback.
-
|
I tested it by loading 1.71. However, the Python scripts call the latest E3SM-unified version. So, there is some inconsistency between the E3SM-unified we source in the beginning the version used by the Python scripts in zppy. I ran into the same issue using both 1.71. and 1.81. |
Beta Was this translation helpful? Give feedback.
-
|
@araman2 You can change what environment commands you're using via the |
Beta Was this translation helpful? Give feedback.
-
|
Moved to discussions since this a user question. In In I will be doing testing on Compy as part of my testing for E3SM-Unified (1.8.0rc1), so I will see if I run into any issues myself. |
Beta Was this translation helpful? Give feedback.
-
|
I was able to generate the global time series plots on Compy. Let me know if running Also, if you want to have the PDF visible on the web immediately, you'd need to set |
Beta Was this translation helpful? Give feedback.
-
|
Thanks much. I tested zppy using the latest released version in cori. Treht json files show negative values. Is this the case for your runs too? Also, in the global time series, some variables like change in ocean heat content and change in sea level are empty in the outputs |
Beta Was this translation helpful? Give feedback.
-
See @chengzhuzhang's response on #356. This should be resolved in the upcoming release.
Are you running |
Beta Was this translation helpful? Give feedback.
-
|
@forsyth2 Thanks. I'm not running mpas_analysis. I want to follow up on the www directory in compyfs. My code is located here: /compyfs/rama225/v2.LR.abrupt-1970-aerosol_nco5xSO4SO2_0301/zppy_scripts/zppy_5xSO4_v2.cfg Running zppy creates netcdf files but doesn't create json or pdf files in the www directory. I have the /compyfs/www/rama225 directory and the subdirectories for zppy outputs as listed in the code. |
Beta Was this translation helpful? Give feedback.
-
|
@araman2 I ran a version of your My |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for trying this.
https://compy-dtn.pnl.gov/fors729/zppy_support_n348v2/v2.LR.abrupt-1970-aerosol_nco5xSO4SO2_0301_production/e3sm_diags/atm_monthly_180x360_aave/model_vs_obs_0301-0450/area_mean_time_series/TREFHT/TREFHT.png
Although it produces pdf plots, the figures have negative values for
TREFHT even in this version of e3smunified.
Aish
…On Wed, Nov 23, 2022 at 4:28 PM forsyth2 ***@***.***> wrote:
@araman2 <https://github.com/araman2> I ran a version of your cfg with
the latest Unified Release Candidate (source /share/apps/E3SM/conda_envs/
test_e3sm_unified_1.8.0rc5_compy.sh). I see the pdf created in
https://compy-dtn.pnl.gov/fors729/zppy_support_n348v2/v2.LR.abrupt-1970-aerosol_nco5xSO4SO2_0301_production/global_time_series/global_time_series_0301-0450_results/v2.LR.5xSO4.pdf
and the json files in
https://compy-dtn.pnl.gov/fors729/zppy_support_n348v2/v2.LR.abrupt-1970-aerosol_nco5xSO4SO2_0301_production/e3sm_diags/atm_monthly_180x360_aave/model_vs_obs_0301-0450/area_mean_time_series/FLUT/
(and the other subdirectories)
My cfg:
[default]
input = /compyfs/rama225/v2.LR.abrupt-1970-aerosol_nco5xSO4SO2_0301
input_subdir = run
#output = /compyfs/rama225/v2.LR.abrupt-1970-aerosol_nco5xSO4SO2_0301/zppy_output
output = /compyfs/fors729/zppy_support_n348v2
case = v2.LR.abrupt-1970-aerosol_nco5xSO4SO2_0301_production
#www = /compyfs/www/rama225/v2.LR.abrupt-1970-aerosol_nco5xSO4SO2_0301/zppy_output
www = /compyfs/www/fors729/zppy_support_n348v2
partition = short
campaign = "water_cycle"
#environment_commands = "source /share/apps/E3SM/conda_envs/load_e3sm_unified_1.6.0_compy.sh"
environment_commands = "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.8.0rc5_compy.sh"
# Regridded atmosphere climatologies every 20 and 50 years
[climo]
active = True
years = "301:450:5", "301:450:20", "301:450:50",
# Use default vars
# Monthly climatology
[[ atm_monthly_180x360_aave ]]
mapping_file = /qfs/people/zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc
frequency = "monthly"
# Diurnal climatology (specifically, 3-hour increments of a day averaged over every day of a month)
# [[ atm_monthly_diurnal_8xdaily_180x360_aave ]]
# input_subdir = "archive/atm/hist"
# input_files = "eam.h4"
# mapping_file = /global/homes/z/zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc
# vars = "PRECT"
# frequency = "diurnal_8xdaily"
# Regridded atmosphere monthly time series files
[ts]
active = True
#environment_commands = "source /share/apps/E3SM/conda_envs/load_e3sm_unified_1.6.0_compy.sh"
years = "301:450:150",
# Monthly time series
[[ atm_monthly_180x360_aave ]]
input_subdir = "run"
input_files = "eam.h0"
frequency = "monthly"
mapping_file = '/qfs/people/zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc'
# Use default vars
# # Daily time series
# [[ atm_daily_180x360_aave ]]
# input_subdir = "archive/atm/hist"
# input_files = "eam.h1"
# frequency = "daily"
# mapping_file = /home/ac.zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc
# vars = "PRECT"
# Monthly global time series
[[ atm_monthly_glb ]]
input_subdir = "run"
input_files = "eam.h0"
frequency = "monthly"
mapping_file = "glb"
# Use default vars
# # Monthly land time series
# [[ land_monthly ]]
# input_subdir = "archive/lnd/hist"
# input_files = "elm.h0"
# frequency = "monthly"
# mapping_file = /home/ac.zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc
# vars = "FSH,RH2M"
# [[ rof_monthly ]]
# input_subdir = "archive/rof/hist"
# input_files = "mosart.h0"
# frequency = "monthly"
# mapping_file = ""
# vars = "RIVER_DISCHARGE_OVER_LAND_LIQ"
# extra_vars = 'areatotal2'
[tc_analysis]
active = False
years = "1:100:20", "1:100:50",
scratch = <scratch directory>
# E3SM Diags every 20 and 50 years
[e3sm_diags]
active = True
#environment_commands = "source /share/apps/E3SM/conda_envs/load_e3sm_unified_1.6.0_compy.sh"
years = "301:450:150",
ts_num_years = 150
# # Monthly diagnostics
# [[ atm_monthly_180x360_aave ]]
# short_name = '20210122.v2_test01.piControl.ne30pg2_EC30to60E2r2-1900_ICG.chrysalis'
# grid = '180x360_aave'
# reference_data_path = '/lcrc/soft/climate/e3sm_diags_data/obs_for_e3sm_diags/climatology'
# # These two lines need to be included if sets requiring time series files are being run.
# obs_ts = '/lcrc/soft/climate/e3sm_diags_data/obs_for_e3sm_diags/time-series'
# ref_start_yr = 1979
# # This needs to be set for QBO Diags
# ref_final_yr = 2016
# # This needs to be set for streamflow diags
# streamflow_obs_ts = '/lcrc/group/e3sm/public_html/e3sm_diags_test_data/unit_test_complete_run/obs/time-series/'
# # These three lines need to be included if diurnal cycle diagnostics are being run.
# dc_obs_climo = '/lcrc/group/e3sm/public_html/e3sm_diags_test_data/unit_test_complete_run/obs/climatology'
# climo_diurnal_subsection = "atm_monthly_diurnal_8xdaily_180x360_aave"
# climo_diurnal_frequency = "diurnal_8xdaily"
# # This is for tropical cyclone diags
# tc_obs = '/lcrc/group/e3sm/public_html/diagnostics/observations/Atm/tc-analysis/'
# Monthly diagnostics
[[ atm_monthly_180x360_aave ]]
short_name = 'v2.LR.5xSO4SO2_0301'
sets="area_mean_time_series",
grid = '180x360_aave'
## reference_data_path = '/lcrc/soft/climate/e3sm_diags_data/obs_for_e3sm_diags/climatology'
# These two lines need to be included if sets requiring time series files are being run.
## obs_ts = '/lcrc/soft/climate/e3sm_diags_data/obs_for_e3sm_diags/time-series'
ref_start_yr = 1850
ref_names = "none", ## to turn off comparison to (nonexistent) obs time series
# This needs to be set for QBO Diags
# ref_final_yr = 2016
# This needs to be set for streamflow diags
# streamflow_obs_ts = '/lcrc/group/e3sm/public_html/e3sm_diags_test_data/unit_test_complete_run/obs/time-series/'
# These three lines need to be included if diurnal cycle diagnostics are being run.
# dc_obs_climo = '/lcrc/group/e3sm/public_html/e3sm_diags_test_data/unit_test_complete_run/obs/climatology'
# climo_diurnal_subsection = "atm_monthly_diurnal_8xdaily_180x360_aave"
# climo_diurnal_frequency = "diurnal_8xdaily"
# This is for tropical cyclone diags
# tc_obs = '/lcrc/group/e3sm/public_html/diagnostics/observations/Atm/tc-analysis/'
[amwg]
active = False
# MPAS-Analysis
[mpas_analysis]
active = False
walltime = "24:00:00"
parallelTaskCount = 6
ts_years = "1-50", "1-100",
enso_years = "11-50", "11-100",
climo_years = "21-50", "51-100",
mesh = "EC30to60E2r2"
# Global time series plots
[global_time_series]
active = True
atmosphere_only=True
#environment_commands = "source /share/apps/E3SM/conda_envs/load_e3sm_unified_1.6.0_compy.sh"
years = "301-450",
ts_num_years = 150
figstr=v2.LR.5xSO4
moc_file=mocTimeSeries_0001-0099.nc
experiment_name=v2.LR.5xSO4
#ts_years = "301-450"
climo_years = "321-350", "351-450",
—
Reply to this email directly, view it on GitHub
<#348 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AI5XFNHPSUO2WP3ZQBNKC3TWJ2ZDXANCNFSM6AAAAAAR3YBNKM>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
--
*Aishwarya Raman, Ph.D. *
Postdoctoral Researcher
Pacific NorthWest National Lab
Richland, WA
***@***.***
|
Beta Was this translation helpful? Give feedback.
-
|
Hey Aish, there might be a bug in e3sm_diags. Do you have some lat-lon maps plots that I can look at? |
Beta Was this translation helpful? Give feedback.
-
|
@araman2 I do find that the time series plots for TREFHT from the global_time_series task of zppy provide reasonable results: https://compy-dtn.pnl.gov/rama225/v2.LR.abrupt-1970-aerosol_nco5xSO4SO2_0301/zppy_output/v2.LR.abrupt-1970-aerosol_nco5xSO4SO2_0301_production/global_time_series/global_time_series_0301-0450_results/v2.LR.5xSO4.pdf As a side note, you could turn on other diagnostics in e3sm_diags to look at other sets. |
Beta Was this translation helpful? Give feedback.
-
|
@chengzhuzhang thanks for taking a look at the plots and the e3sm_diags code. If I use e3sm-unified v1.8.1 or 1.7, area mean plots show negative values of TREHFT. For example, this was created using the latest e3sm-unified version. The timeseries issue I'm experiencing is with v1.5.1 which shows reasonable TREFHT in area mean plots but does not plot global timeseries because of the error in interpreting years. Details are here: |
Beta Was this translation helpful? Give feedback.
-
|
@chengzhuzhang Thanks for taking a look at the plots and the e3sm_diags code. There are separate issues while using v1.8.1 (or 1.7, 1.6) and v1.51 of e3sm-unified. If I use e3sm-unified v1.8.1 or 1.7, area mean plots show negative values of TREHFT. For example, this was created using the latest e3sm-unified version. The timeseries issue I'm experiencing is with v1.5.1 which shows reasonable TREFHT in area mean plots but does not plot global timeseries because of the error in interpreting years. Details are here: #356 |
Beta Was this translation helpful? Give feedback.
-
|
Hi @araman2 Thanks for the added information. I'm tagging @xylar for reporting this issue. Hey Xylar, it doesn't look like 1.8.1 addressed the remapping issue that we saw in 1.7.1 on compy. |
Beta Was this translation helpful? Give feedback.
-
|
@chengzhuzhang and @araman2, I'm sorry that this isn't fixed in E3SM-Unified 1.8.0 (there isn't a 1.8.1 yet and hopefully we won't have one). I don't have any relevant expertise to help fix this issue and @czender doesn't have access to Compy as far as I know. This leaves us with relatively few options to debug the problem and help. @rljacob, is there anything E3SM can do to smooth the process for @czender getting access? This is becoming a problem for our ability to support analysis on Compy at all. |
Beta Was this translation helpful? Give feedback.
-
|
@xylar If i remembered correctly, I think @czender actually got the Compy access.. |
Beta Was this translation helpful? Give feedback.
-
|
@okay, in that case I think he's the person to ping. |
Beta Was this translation helpful? Give feedback.
-
|
Hi @czender one example file with this issue can be found here: /compyfs/fors729/zppy_support_n348v2/post/atm/180x360_aave/clim/50yr/v2.LR.abrupt-1970-aerosol_nco5xSO4SO2_0301_production_ANN_030101_035012_climo.nc The input file is here:/compyfs/rama225/v2.LR.abrupt-1970-aerosol_nco5xSO4SO2_0301/run But I suppose this behavior can be reproduced on those standard v2 atm files with e3sm_unified_1.8.0rc on Compy. I recall that ncclimo in e3sm_unified_1.6.0 worked fine on Compy. Some changes between 1.6.0 to 1.7.0 caused this issue. We need expertise from both @czender and @xylar for troubleshooting this problem. |
Beta Was this translation helpful? Give feedback.
-
|
An important change between 1.6.0 and 1.7.x was that I switched from building with intel on compy to gnu because (if memory servers) I couldn't get NCO or another dependency to build with intel at all. I expected NCO to be better behaved with gnu than intel on compy anyway because my sense is that more testing happens with gnu (that is what conda-forge uses, for example). |
Beta Was this translation helpful? Give feedback.
-
|
Hi All, yes my Compy account was restored about two months ago. Then I forgot to look into this issue. I will try to do so this week... |
Beta Was this translation helpful? Give feedback.
-
|
FYI After many star-crossed attempts, I have (finally) reproduced this issue on Compy with E3SMU-1.8.0. The debugging will begin in earnest now... |
Beta Was this translation helpful? Give feedback.
-
|
Thank you Charlie @czender ! Hope this is not something supper stubborn.. |
Beta Was this translation helpful? Give feedback.
-
|
FYI I have fixed this issue in NCO 5.1.4-alpha06. The cause was related to an OpenMP SIMD pragma that behaves differently, for some reason, on Compy compute nodes with Spack-built executables than anywhere else compiled any other way. I'll tag NCO 5.1.4-final by next week. That will likely be the version to include in 1.8.1-final. In the meantime, please anyone who is affected by this issue verify that the fix works for you by testing E3SMU 1.8.1-rc2 or later on Compy with |
Beta Was this translation helpful? Give feedback.
-
|
Tagging @quaz115, @araman2, @kaizhangpnl, @BunnyVon, I think you reported or potentially be impacted by this issue on Compy. When you have a chance, please follow the instruction given by Charlie above to try the new e3sm_unified version. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks, Charlie and Jill. I can try it next week and let you know.
Thanks,
Aish.
On Fri, Jan 6, 2023 at 9:53 AM Jill Chengzhu Zhang ***@***.***> wrote:
Tagging @quaz115 <https://github.com/quaz115>, @araman2
<https://github.com/araman2>, @kaizhangpnl
<https://github.com/kaizhangpnl>, @BunnyVon <https://github.com/BunnyVon>,
I think you reported or potentially be impacted by this issue on Compy.
When you have a chance, please follow the instruction given by Charlie
above to try the new e3sm_unified version.
—
Reply to this email directly, view it on GitHub
<#348 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AI5XFNBAZIOTUOGZS2DYFITWRBLYFANCNFSM6AAAAAAR3YBNKM>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
--
*Aishwarya Raman, Ph.D. *
Postdoctoral Researcher
Pacific NorthWest National Lab
Richland, WA
***@***.***
|
Beta Was this translation helpful? Give feedback.
-
Running zppy in compy does not create .pdf or .png files and TREFHT < 0. Log files or status files do not show specific error messages that hinder the creation of .pdf outputs.
zppy outputs for the simulation are located here: /compyfs/rama225/v2.LR.abrupt-1970-aerosol_0301/zppy_output/post/scripts
zppy config file for the outputs are located here: /compyfs/rama225/v2.LR.abrupt-1970-aerosol_0301/zppy_scripts/zppy_abrupt_aerosol_1970.cfg
Tried zppy with both the latest e3sm unified and the previous versions in compy.
Beta Was this translation helpful? Give feedback.
All reactions