Move the variational scripts to the g-w #1282

guillaumevernieres · 2024-09-13T17:55:53Z

This PR needs to be merged before

Move the variational scripts to this repository global-workflow#2920

It depends on

Revive the marine jcb implementation jcb-gdas#26

and will require an update to the jcb-gdas submodules.

It will break the GDASApp marine CI until the g-w pr is merged.

AndrewEichmann-NOAA

This looks very satisfying

apchoiCMD

Looks great!

emcbot · 2024-09-17T22:55:42Z

Automated Global-Workflow GDASApp Testing Results:
Machine: hera

Start: Tue Sep 17 21:08:15 UTC 2024 on hfe04
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Tue Sep 17 21:55:05 UTC 2024
---------------------------------------------------
Tests:                                  *Failed*
Tests: Failed at Tue Sep 17 22:55:41 UTC 2024
Tests: 84% tests passed, 6 tests failed out of 37
	1902 - test_gdasapp_atm_jjob_var_init (Failed)
	1903 - test_gdasapp_atm_jjob_var_run (Failed)
	1904 - test_gdasapp_atm_jjob_var_inc (Failed)
	1905 - test_gdasapp_atm_jjob_var_final (Failed)
	1906 - test_gdasapp_atm_jjob_ens_init (Failed)
Tests: see output at /scratch1/NCEPDEV/da/role.jedipara/CI/GDASApp/workflow/PR/1282/global-workflow/sorc/gdas.cd/build/log.ctest

emcbot · 2024-09-18T01:56:37Z

Automated Global-Workflow GDASApp Testing Results:
Machine: hera

Start: Wed Sep 18 01:38:57 UTC 2024 on hfe04
---------------------------------------------------
Build:                                  *FAILED*
Build: Failed at Wed Sep 18 01:56:36 UTC 2024
Build: see output at /scratch1/NCEPDEV/da/role.jedipara/CI/GDASApp/workflow/PR/1282/global-workflow/sorc/log.build

emcbot · 2024-09-18T03:59:49Z

Automated Global-Workflow GDASApp Testing Results:
Machine: hera

Start: Wed Sep 18 03:42:26 UTC 2024 on hfe12
---------------------------------------------------
Build:                                  *FAILED*
Build: Failed at Wed Sep 18 03:59:48 UTC 2024
Build: see output at /scratch1/NCEPDEV/da/role.jedipara/CI/GDASApp/workflow/PR/1282/global-workflow/sorc/log.build

RussTreadon-NOAA · 2024-09-18T10:03:42Z

Compare /scratch1/NCEPDEV/da/role.jedipara/CI/GDASApp/workflow/PR/1282/global-workflow/sorc/logs/build_gdas.log with /scratch1/NCEPDEV/da/role.jedipara/git/global-workflow/radbcor/sorc/logs/build_gdas.log.

The PR/1282 build_gdas.log contains

-- Adding bundle project gdas
-- ---------------------------------------------------------
-- [GDASApp] (1.0.0)
-- Feature TESTS enabled
-- Found Python3: /scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.6.0/envs/unified-env-rocky8/install/intel/2021.5.0/python-3.10.13-2p2rady/bin/python3.10 (found version "3.10.13") found components: Interpreter
CMake Error at gdas/test/gw-ci/CMakeLists.txt:7 (set_tests_properties):
  set_tests_properties Can not find test to add properties to:
  WCDA-3DVAR-C48mx500
Call Stack (most recent call first):
  gdas/test/gw-ci/CMakeLists.txt:62 (add_cycling_tests)

whereas radbcor build_gdas.log has

-- Adding bundle project gdas
-- ---------------------------------------------------------
-- [GDASApp] (1.0.0)
-- Feature TESTS enabled
-- Found Python3: /scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.6.0/envs/unified-env-rocky8/install/intel/2021.5.0/python-3.10.13-2p2rady/bin/python3.10 (found version "3.10.13") found components: Interpreter
ctest for JGLOBAL_PREP_OCEAN_OBS
ctest for JGLOBAL_MARINE_BMAT
ctest for JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP
ctest for JGDAS_GLOBAL_OCEAN_ANALYSIS_RUN
ctest for JGDAS_GLOBAL_OCEAN_ANALYSIS_CHKPT
ctest for JGDAS_GLOBAL_OCEAN_ANALYSIS_POST
-- ---------------------------------------------------------

I see the comment It will break the GDASApp marine CI until the g-w pr is merged. above.

Does this mean the hera-GW-RT failure is expected until g-w PR #2920 is merged?
Until then, do we need to run GDASApp CI inside a working copy of the PR #2920 branch, guillaumevernieres:feature/marinevar?

guillaumevernieres · 2024-09-18T12:27:03Z

Thanks for checking @RussTreadon-NOAA . The CI for this PR should work, it's built/tested against the g-w feature/marinevar branch from NOAA-EMC/global-workflow/pull/2920.
I'll fix the issue this am.

… into feature/marinevar

JohnSteffen-NOAA

Looks good.

apchoiCMD

Looks promising!

guillaumevernieres · 2024-09-18T15:40:17Z

1/2 🎉

...
      Start 1890: test_gdasapp_WCDA-3DVAR-C48mx500
23/46 Test #1890: test_gdasapp_WCDA-3DVAR-C48mx500 ...................................   Passed    9.41 sec
      Start 1891: test_gdasapp_WCDA-3DVAR-C48mx500_gdasstage_ic_202103241200
24/46 Test #1891: test_gdasapp_WCDA-3DVAR-C48mx500_gdasstage_ic_202103241200 .........   Passed  1095.50 sec
      Start 1892: test_gdasapp_WCDA-3DVAR-C48mx500_gdasfcst_202103241200
25/46 Test #1892: test_gdasapp_WCDA-3DVAR-C48mx500_gdasfcst_202103241200 .............   Passed  432.87 sec
      Start 1893: test_gdasapp_WCDA-3DVAR-C48mx500_gdasprepoceanobs_202103241800
26/46 Test #1893: test_gdasapp_WCDA-3DVAR-C48mx500_gdasprepoceanobs_202103241800 .....   Passed  545.51 sec
      Start 1894: test_gdasapp_WCDA-3DVAR-C48mx500_gdasmarinebmat_202103241800
27/46 Test #1894: test_gdasapp_WCDA-3DVAR-C48mx500_gdasmarinebmat_202103241800 .......   Passed  430.77 sec
      Start 1895: test_gdasapp_WCDA-3DVAR-C48mx500_gdasmarineanlinit_202103241800
28/46 Test #1895: test_gdasapp_WCDA-3DVAR-C48mx500_gdasmarineanlinit_202103241800 ....   Passed   81.69 sec
      Start 1896: test_gdasapp_WCDA-3DVAR-C48mx500_gdasmarineanlvar_202103241800
29/46 Test #1896: test_gdasapp_WCDA-3DVAR-C48mx500_gdasmarineanlvar_202103241800 .....   Passed  200.57 sec
      Start 1897: test_gdasapp_WCDA-3DVAR-C48mx500_gdasmarineanlchkpt_202103241800
30/46 Test #1897: test_gdasapp_WCDA-3DVAR-C48mx500_gdasmarineanlchkpt_202103241800 ...   Passed  124.72 sec
      Start 1898: test_gdasapp_WCDA-3DVAR-C48mx500_gdasmarineanlfinal_202103241800
31/46 Test #1898: test_gdasapp_WCDA-3DVAR-C48mx500_gdasmarineanlfinal_202103241800 ...   Passed  106.90 sec
...

Should I ignore the atmos tests @RussTreadon-NOAA ? They are all failing, I hope I didn't delete something I shouldn't have during my rm * frenzy.

RussTreadon-NOAA · 2024-09-18T15:50:41Z

@guillaumevernieres: Yes, we expect atmanlinit and atmensanlinit to fail because GDASApp has been updated to process a radiance bias correction tarball. g-w still assumes we are dealing with satellite_sensor specific radiance bias correction files.

g-w PR #2875 has the fix but this PR has not yet been approved and merged into g-w develop.

I looked at /scratch1/NCEPDEV/da/role.jedipara/CI/GDASApp/workflow/PR/1282/global-workflow/sorc/gdas.cd/build/gdas/test/atm/global-workflow/testrun/atmanlinit-66754880.out and confirmed that the failure is due to the radiance bias correction file disconnect.

emcbot · 2024-09-18T16:37:16Z

Automated Global-Workflow GDASApp Testing Results:
Machine: hera

Start: Wed Sep 18 13:57:19 UTC 2024 on hfe02
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Wed Sep 18 14:43:12 UTC 2024
---------------------------------------------------
Tests:                                  *Failed*
Tests: Failed at Wed Sep 18 16:37:15 UTC 2024
Tests: 87% tests passed, 6 tests failed out of 46
	1902 - test_gdasapp_atm_jjob_var_init (Failed)
	1903 - test_gdasapp_atm_jjob_var_run (Failed)
	1905 - test_gdasapp_atm_jjob_var_final (Failed)
	1906 - test_gdasapp_atm_jjob_ens_init (Failed)
Tests: see output at /scratch1/NCEPDEV/da/role.jedipara/CI/GDASApp/workflow/PR/1282/global-workflow/sorc/gdas.cd/build/log.ctest

guillaumevernieres · 2024-09-18T17:15:42Z

I'm going to merge this, which will break the marine DA ctest. Ignore until NOAA-EMC/global-workflow/pull/2920 is merged.

guillaumevernieres and others added 8 commits August 28, 2024 09:11

...

7c9afcf

obsstats

a6a4529

wip

be72aa2

yaml reorg

d6d442f

wip

4c00bab

Merge branch 'develop' into feature/marinevar

02cfdaa

tidy?

3c3d49f

hide unused

4c87eec

guillaumevernieres mentioned this pull request Sep 13, 2024

Move the variational scripts to this repository NOAA-EMC/global-workflow#2920

Merged

16 tasks

guillaumevernieres added 5 commits September 16, 2024 17:52

renamed obs yaml dir

114af2b

updated jcb hash

38e10f9

got a bit crazy with rm

9ed2dfc

Merge branch 'develop' into feature/marinevar

f5e71b1

Merge branch 'develop' into feature/marinevar

ba6dc96

guillaumevernieres requested review from AndrewEichmann-NOAA, apchoiCMD, JohnSteffen-NOAA, givelberg and KatherineLukens-NOAA September 17, 2024 20:44

guillaumevernieres added the hera-GW-RT Queue for automated testing with global-workflow on Hera label Sep 17, 2024

AndrewEichmann-NOAA reviewed Sep 17, 2024

View reviewed changes

apchoiCMD previously approved these changes Sep 17, 2024

View reviewed changes

emcbot added hera-GW-RT-Running Automated testing with global-workflow running on Hera and removed hera-GW-RT Queue for automated testing with global-workflow on Hera labels Sep 17, 2024

emcbot added hera-GW-RT-Failed Automated testing with global-workflow failed on Hera and removed hera-GW-RT-Running Automated testing with global-workflow running on Hera labels Sep 17, 2024

appended test_gdasapp to marine tests

92ab4ec

guillaumevernieres dismissed apchoiCMD’s stale review via 92ab4ec September 18, 2024 01:19

guillaumevernieres removed the hera-GW-RT-Failed Automated testing with global-workflow failed on Hera label Sep 18, 2024

emcbot added the hera-GW-RT-Running Automated testing with global-workflow running on Hera label Sep 18, 2024

emcbot added hera-GW-RT-Failed Automated testing with global-workflow failed on Hera and removed hera-GW-RT-Running Automated testing with global-workflow running on Hera labels Sep 18, 2024

guillaumevernieres added hera-GW-RT Queue for automated testing with global-workflow on Hera and removed hera-GW-RT-Failed Automated testing with global-workflow failed on Hera labels Sep 18, 2024

emcbot added hera-GW-RT-Running Automated testing with global-workflow running on Hera and removed hera-GW-RT Queue for automated testing with global-workflow on Hera labels Sep 18, 2024

emcbot added hera-GW-RT-Failed Automated testing with global-workflow failed on Hera and removed hera-GW-RT-Running Automated testing with global-workflow running on Hera labels Sep 18, 2024

guillaumevernieres added 2 commits September 18, 2024 08:08

Merge branch 'feature/marinevar' of https://github.com/NOAA-EMC/GDASApp…

cf7d590

… into feature/marinevar

fixed ctest base name

a1c7ec6

guillaumevernieres added hera-GW-RT Queue for automated testing with global-workflow on Hera and removed hera-GW-RT-Failed Automated testing with global-workflow failed on Hera labels Sep 18, 2024

emcbot added hera-GW-RT-Running Automated testing with global-workflow running on Hera and removed hera-GW-RT Queue for automated testing with global-workflow on Hera labels Sep 18, 2024

JohnSteffen-NOAA approved these changes Sep 18, 2024

View reviewed changes

apchoiCMD approved these changes Sep 18, 2024

View reviewed changes

emcbot added hera-GW-RT-Failed Automated testing with global-workflow failed on Hera and removed hera-GW-RT-Running Automated testing with global-workflow running on Hera labels Sep 18, 2024

guillaumevernieres merged commit 55e895f into develop Sep 18, 2024
11 checks passed

guillaumevernieres deleted the feature/marinevar branch September 18, 2024 17:15

RussTreadon-NOAA mentioned this pull request Sep 27, 2024

GDASApp EE2 compliance checklist for GFS v17 & v18 #1254

Open

10 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move the variational scripts to the g-w #1282

Move the variational scripts to the g-w #1282

guillaumevernieres commented Sep 13, 2024 •

edited

Loading

AndrewEichmann-NOAA left a comment

apchoiCMD left a comment

emcbot commented Sep 17, 2024

emcbot commented Sep 18, 2024

emcbot commented Sep 18, 2024

RussTreadon-NOAA commented Sep 18, 2024

guillaumevernieres commented Sep 18, 2024

JohnSteffen-NOAA left a comment

apchoiCMD left a comment

guillaumevernieres commented Sep 18, 2024

RussTreadon-NOAA commented Sep 18, 2024

emcbot commented Sep 18, 2024

guillaumevernieres commented Sep 18, 2024

Move the variational scripts to the g-w #1282

Move the variational scripts to the g-w #1282

Conversation

guillaumevernieres commented Sep 13, 2024 • edited Loading

AndrewEichmann-NOAA left a comment

Choose a reason for hiding this comment

apchoiCMD left a comment

Choose a reason for hiding this comment

emcbot commented Sep 17, 2024

emcbot commented Sep 18, 2024

emcbot commented Sep 18, 2024

RussTreadon-NOAA commented Sep 18, 2024

guillaumevernieres commented Sep 18, 2024

JohnSteffen-NOAA left a comment

Choose a reason for hiding this comment

apchoiCMD left a comment

Choose a reason for hiding this comment

guillaumevernieres commented Sep 18, 2024

RussTreadon-NOAA commented Sep 18, 2024

emcbot commented Sep 18, 2024

guillaumevernieres commented Sep 18, 2024

guillaumevernieres commented Sep 13, 2024 •

edited

Loading