Small number protections to CCH water transfer functions #1164

rgknox · 2024-02-22T21:09:51Z

A cap has been created on the smallest suction that we are willing to simulate with CCH (Campbell Clapp-Hornberger) water transfer functions. These methods determine the equivalent water content, so that if values below this water content are passed to the functions, it operates at the lowest threshold instead of breaking. The current threshold is -15 MPa. This is incredibly small, and should generate conductances that indistinguishable from conductances generated with near zero water contents.

A further modification was made in a second round of development here. All water retention curves are now designed to have a minimum value. For CCH it is -15 MPa, for Van Genucten it is the residual water content (existing parameter). We now force the fraction of maximum conductivity to be zero at the minimum value, this currently includes stomatal conductance. This is applied by determining a weighting function that blends in a zero value. The weighting is 1 at the minimum suction, and decreases to zero on an exponential. I used an exponent that attenuates to a value of about 0.175 after increasing 1 MPa above minimum, and 0.025 after 2 MPa above minimum. This function makes the derivative of conductance with suction continuous and smooth, but most importantly, it prevents the flow of water in plant and soil media that are completely parched.

This method of ensuring fluxes are zero at these super low water contents will not affect re-hydration, because conductance between nodes is driven by the suction of the up-stream compartment (ie the one with more total potential, or less suction). However, this will prevent completely desicated compartments from going into negative water content.

Description:

Collaborators:

@olyson @jennykowalcz

Expectation of Answer Changes:

This should prevent crashes, particularly in soil water calls using frozen soils.

Fixes: #1168
Fixes: #1163

Checklist

If this is your first time contributing, please read the CONTRIBUTING document.

All checklist items must be checked to enable merging this pull request:

Contributor

The in-code documentation has been updated with descriptive comments
The documentation has been assessed to determine if updates are necessary

Integrator

FATES PASS/FAIL regression tests were run
Evaluation of test results for answer changes was performed and results provided

Documentation

Technical Note update:
User's Guide update:

Test Results:

CTSM (or) E3SM (specify which) test hash-tag:

CTSM (or) E3SM (specify which) baseline hash-tag:

FATES baseline hash-tag:

Test Output:

…ns so that it will tolerate incredibly small water contents

…ection

rgknox · 2024-02-26T18:15:44Z

This may address: #1154

jennykowalcz · 2024-02-26T19:46:12Z

This may address: #1154

I tried this out -- in the control case, without this PR, the case fails with this error:

128: PIO: FATAL ERROR: Aborting... An error occured, Writing variables (number of variables = 425) to file (./hydro_nocomp
_2pft_campbell_control.elm.h0.0002-11.nc, ncid=141) using PIO_IOTYPE_PNETCDF iotype failed. Non blocking write for variabl
e (FATES_ROOTWGT_SOILMATPOT, varid=274) failed (Number of subarray requests/regions=1, Size of data local to this process 
= 288). NetCDF: Numeric conversion not representable (err=-60). Aborting since the error handler was set to PIO_INTERNAL_E
RROR... (/global/u2/j/jkowalcz/E3SM-test/E3SM/externals/scorpio/src/clib/pio_darray_int.c: 395)

In the test case, pulling in the changes to FatesHydroWTFMod.F90, it fails with this error:

 27:  Site plant water balance does not close
 27:  delta plant storage:  -4.101476403268849E-005  [kg/m2]
 27:  integrated root flux:  -2.537539597419707E-002  [kg/m2]
 27:  transpiration flux:   3.019638043594088E-005  [kg/m2]
 27:  end storage:   1.310177109332794E-002
 27:  pre_h2oveg  1.314278585736063E-002
 27:  ENDRUN:
 27:  ERROR in FatesPlantHydraulicsMod.F90 at line 2808

In both cases I'm using this E3SM branch with this FATES branch (with the change to FatesHydroWTFMod for the test case). I can try also testing with this PR and the hydro stability PR.

The region used in these runs is a lat/lon rectangle from 15 N - 25 S and 30 W - 85 W so it includes very dry regions outside the Amazon basin where FATES-Hydro is unhappy.

glemieux · 2024-03-06T23:42:14Z

Running regression tests on derecho I found that while this resolves #1163, I'm seeing a run failure due to the 1D hydro solver failing with a large wb_step_error:

2461  Could not find a stable solution for hydro 1D solve
2462 
2463  error code:            1
2464  error diag:    0.0000000000000000        0.0000000000000000        0.0000000000000000        0.0000000000000000        0.0000000000000000
2465  lat:   -7.0000000000000000      longitidue:   305.00000000000000
2466  is recruitment:  F
2467  layer:            7
2468  wb_step_err =    1.6512024553098321E+139
2469  q_top_eff*dt_step =    0.0000000000000000
2470  w_tot_beg =    102.64162586485499
2471  w_tot_end =    1.6512024553098321E+139
2472  leaf water:    6.3082104539614020E-002  kg/plant
2473  stem_water:    5.2044616695225525E-002  kg/plant
2474  troot_water:    1.6178663624130458E-002
2475  aroot_water:    1.8182979624968509E-002
2476  LWP:   -2.1266161551233220
2477  dbh:   0.40361081540638905
2478  pft:            6
2479  z nodes:    1.2616446404957407       0.60582232024787031      -0.19099388737231493      -0.79999999999999993      -0.79999999999999993
2480  psi_z:    1.2364117476858283E-002   5.9370587384291323E-003  -1.8717400962486863E-003  -7.8399999999999997E-003   0.0000000000000000
2481  vol,    theta,   H,  Psi,     kmax-
2482  flux:             0.0000000000000000
2483  l:   8.1613119481785556E-005  0.77294073477601488        3.3911760124701393        3.3788118949932811
2484                           4.3843160661903065E-004
2485  s:   8.0363738039128391E-005  0.64761318929546885       -3.6111596928010109E-002  -4.2048655666439241E-002
2486                           1.2058111921054136E-003
2487  t:   2.4966046503811633E-005  0.64802665578870955       -3.6564678574268254E-002
2488                           8.0381337022338061E-004
2489  a:   1.3824320409604611E-006  0.75035604020636570       -2.2154089112156015E-003
2490                     in:   1.6033112452206815E-003
2491                    out:   3.0709050878900186E-006
2492  r1:  0.21880148501536775       0.46850359885155290        6.9272904988199870E+156

I'm guessing that this is due to something causing dth_node to blow up within Hydraulics_Tridiagonal?

Note that the log stops writing early due to #1168, but I'm not entirely sure if that error is actually due to the above error.

jennykowalcz · 2024-03-07T14:31:05Z

This may address: #1154

I tried this out -- in the control case, without this PR, the case fails with this error:
128: PIO: FATAL ERROR: Aborting... An error occured, Writing variables (number of variables = 425) to file (./hydro_nocomp
_2pft_campbell_control.elm.h0.0002-11.nc, ncid=141) using PIO_IOTYPE_PNETCDF iotype failed. Non blocking write for variabl
e (FATES_ROOTWGT_SOILMATPOT, varid=274) failed (Number of subarray requests/regions=1, Size of data local to this process 
= 288). NetCDF: Numeric conversion not representable (err=-60). Aborting since the error handler was set to PIO_INTERNAL_E
RROR... (/global/u2/j/jkowalcz/E3SM-test/E3SM/externals/scorpio/src/clib/pio_darray_int.c: 395)
In the test case, pulling in the changes to FatesHydroWTFMod.F90, it fails with this error:
 27:  Site plant water balance does not close
 27:  delta plant storage:  -4.101476403268849E-005  [kg/m2]
 27:  integrated root flux:  -2.537539597419707E-002  [kg/m2]
 27:  transpiration flux:   3.019638043594088E-005  [kg/m2]
 27:  end storage:   1.310177109332794E-002
 27:  pre_h2oveg  1.314278585736063E-002
 27:  ENDRUN:
 27:  ERROR in FatesPlantHydraulicsMod.F90 at line 2808  
In both cases I'm using this E3SM branch with this FATES branch (with the change to FatesHydroWTFMod for the test case). I can try also testing with this PR and the hydro stability PR.

The region used in these runs is a lat/lon rectangle from 15 N - 25 S and 30 W - 85 W so it includes very dry regions outside the Amazon basin where FATES-Hydro is unhappy.

Unfortunately I also get this error with a domain restricted to the Amazon Basin and when including @xuchongang's hydro stability branch. Also, I forgot to mention that I am using fates_hydro_solver = 2 .

glemieux · 2024-03-07T16:33:50Z

I should note that the regression tests for this were all b4b otherwise.

derecho location: /glade/u/home/glemieux/scratch/ctsm-tests/tests_pr1164-fates-correctbase

mpaiao · 2024-03-07T17:16:19Z

biogeophys/FatesHydroWTFMod.F90

-        th = this%th_sat*(psi/this%psi_sat)**(-1.0_r8/this%beta)
+     else
+        if(psi<this%psi_min) then
+           th = this%th_min


Wouldn't this artificially add water to soils?

yes, but the intention is to go from a value of zero to an inconsequentially small non-zero value. @ckoven and I were talking, this could potentially be problematic if we artificially add small amounts of water and that enables or drives non-zero fluxes that would had otherwise been zero. One possibility would be to force fractional conductivity to zero where it hits theta min. That seems messy though. Honestly, I think if we just use a suitably small minimum value, one that is an incredibly small water content, yet can still accomodate the math needed to switch between psi and theta, this should be sufficient. Perhaps we can try minimum capping at -15 MPa or smaller, perhaps -9 is too high.

Will this trigger mass balance check error if we run the model for a long time (let's say 1000 years for spinup) ? Should we should keep track of this error and used it as part of the mass balance error?

I don't believe this is a mass conservation liability. We use these functions to drive fluxes, they never actually over-write the original masses of each pool. It is the fluxes that update the pools. What will happen is that water contents below the psi_min or theta_min values will just end up having zero flux out of that pool, and thus remain unchanged.

For instance here is where we update the actual soil water content, it is based off of fluxes:
https://github.com/NGEET/fates/blob/sci.1.74.0_api.35.0.0/biogeophys/FatesPlantHydraulicsMod.F90#L2729-L2730

…defined variables, reduce minimum suction of cch and fixed an error check

Bug fix. Adrianna Foster identified that trunks should be removed from the patch-level variable sum_fuel before being used to calculate rate of spread and fuel consumption.

…luxes to soil layer fluxes when no conductance (trivial bug fix).

rgknox · 2024-04-17T00:49:27Z

These changes seem to be adding some stability. I ran a 25 year smoke test on the F45 grid with ELM-FATES-Hydro. Vegetation was initialized as a cold start using the specified stem diameter (which I specified as the diameter at 75% maximum height) and closed canopy assumption, along with fixed biogeography and no-competition assumptions. All parameters were defaults, with the one exception being the use of the Picard hydraulics solver.

The model completed the smoke test. I show some mean fluxes and states from the final year of simulation. I interpret from the output that the hydraulics model continued to generate fluxes within the realm of realism through the final year of simulation. Note that contrary to the Picard solver, the 1D Taylor solve (default) did crash after several years, having trouble generating a solution within the error tolerance.
f45-hydrotest-biogeonocomp-yr23.pdf

rgknox · 2024-04-17T00:50:24Z

I think this PR is ready for another round of review, and has changed significantly since the first round. I plan no further changes beyond what comes out of review.

jennykowalcz · 2024-04-17T15:11:02Z

Wow, this is great @rgknox ! When perlmutter comes back up I'll try rerunning my various failed hydro cases with this branch and hopefully close several of my hydro Issues.
Should we make the Picard 2D solver the default?

rgknox · 2024-04-17T18:19:59Z

I'm in favor of making Picard the default. I'm curious if others have found similar things. I'll create a discussion!

jennykowalcz · 2024-04-22T15:32:53Z

Wow, this is great @rgknox ! When perlmutter comes back up I'll try rerunning my various failed hydro cases with this branch and hopefully close several of my hydro Issues. Should we make the Picard 2D solver the default?

This PR with fates_hydro_solver=2 allows the cases to run successfully. Like you found, with fates_hydro_solver=1, we still get "Could not find a stable solution for hydro 1D solve" errors in dry gridcells

bchristo · 2024-04-22T19:12:33Z

Chiming in here. This is fantastic news and sounds to me like the Picard solver should be the default.

…

On Mon, Apr 22, 2024, 10:33 AM Jennifer Kowalczyk (Jenny) < ***@***.***> wrote: Wow, this is great @rgknox <https://github.com/rgknox> ! When perlmutter comes back up I'll try rerunning my various failed hydro cases with this branch and hopefully close several of my hydro Issues. Should we make the Picard 2D solver the default? This PR with fates_hydro_solver=2 allows the cases to run successfully. Like you found, with fates_hydro_solver=1, we still get "Could not find a stable solution for hydro 1D solve" errors in dry gridcells — Reply to this email directly, view it on GitHub <#1164 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADSXQZRRO2RVUUSOQBUKPTTY6UUUFAVCNFSM6AAAAABDVVGDSCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANRZHEYTGMBXG4> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

bchristo · 2024-04-23T15:46:19Z

biogeophys/FatesHydroWTFMod.F90

@@ -44,7 +50,8 @@ module FatesHydroWTFMod
                                            ! elastic-caviation region


-  real(r8), parameter :: min_theta_cch = 0.01_r8 ! Minimum theta (matches ctsm)
+  !real(r8), parameter :: min_theta_cch = 0.05_r8 ! Minimum theta (matches ctsm)
+  real(r8), parameter :: min_psi_cch = -20.0_r8 ! Minimum theta (matches ctsm)


update comment to say 'Minimum psi'

bchristo · 2024-04-23T16:26:50Z

biogeophys/FatesHydroWTFMod.F90

+    !this%th_min      = min_theta_cch
+    !this%psi_min     = this%psi_from_th(min_theta_cch+tiny(this%th_max))
+    this%psi_min     = min_psi_cch
+    this%th_min      = this%th_from_psi(min_psi_cch+tiny(this%th_max))


what's the purpose of the tiny(this$th_max)? Wondering if this is an error, since the input to this function should be only water potential.

originally, this tiny() function was used to prevent a problem with order of operations. Inside these functions we have "if statements" that compare theta and psi to these very psi_min and th_min thresholds we are calculating. By adding or subtracting a tiny, we were preventing the intering of logical blocks that were not yet initialized. But with these changes, we actually don't really use these tiny's as much

Understand, but does seems to be a little confusing. Maybe we can clean up the codes in later pull requests.

bchristo · 2024-04-23T16:41:09Z

biogeophys/FatesHydroWTFMod.F90

+    real(r8) :: pc
+    real(r8) :: kr
+    real(r8) :: dkr_dP
+    real(r8) :: sat_res
+    real(r8) :: alpha
+    real(r8) :: lambda
+    real(r8) :: Se
+    real(r8) :: deltaPc
+    real(r8) :: dSe_dpc
+    real(r8) :: dkr_dSe


Just a minor comment. My memory is a bit fuzzy on my Brooks-Corey model without looking it up; can comments on these variables be made to ease interpretability? And an overall comment citing a companion source for implementing the three main different regimes (full, smoothing, saturated).

I agree that it would be great to add comments on the units and meaning of each variables

I created an issue here: #1193
I would just add the changes in here, but I'm not familiar with that model.

xuchongang

I finished my review. It looks great to me (on minimum water content and forced zero conductance) and I think it should resolve the issue of very dry conditions.

xuchongang · 2024-04-25T19:41:15Z

biogeophys/FatesHydroWTFMod.F90

@@ -44,6 +44,8 @@ module FatesHydroWTFMod
                                            ! elastic-caviation region


+  real(r8), parameter :: min_psi_cch = -9._r8 ! Minimum psi we are willing to track in cch


can you put the unit here (Mpa)? Why 9 instead of 15?

I think this was actually updated, seems like a github glitch

xuchongang · 2024-04-25T19:45:18Z

biogeophys/FatesHydroWTFMod.F90

+    !this%th_min      = min_theta_cch
+    !this%psi_min     = this%psi_from_th(min_theta_cch+tiny(this%th_max))
+    this%psi_min     = min_psi_cch
+    this%th_min      = this%th_from_psi(min_psi_cch+tiny(this%th_max))


Understand, but does seems to be a little confusing. Maybe we can clean up the codes in later pull requests.

xuchongang · 2024-04-25T19:48:24Z

biogeophys/FatesHydroWTFMod.F90

-        th = this%th_sat*(psi/this%psi_sat)**(-1.0_r8/this%beta)
+     else
+        if(psi<this%psi_min) then
+           th = this%th_min


Will this trigger mass balance check error if we run the model for a long time (let's say 1000 years for spinup) ? Should we should keep track of this error and used it as part of the mass balance error?

xuchongang · 2024-04-25T19:53:41Z

biogeophys/FatesHydroWTFMod.F90

@@ -44,6 +50,9 @@ module FatesHydroWTFMod
                                            ! elastic-caviation region


+  !real(r8), parameter :: min_theta_cch = 0.05_r8 ! Minimum theta (matches ctsm)
+  real(r8), parameter :: min_psi_cch = -15.0_r8 ! Minimum theta (matches ctsm)


The comment should be minimum water potential (MPa)

xuchongang · 2024-04-26T18:34:10Z

biogeophys/FatesHydroWTFMod.F90

+
+    ! If the difference between psi and psi-min is greater than 10MPa
+    ! just assume there is no effect of the minimum function (ie weight 0)
+    min_ftc_weight = exp(min_ftc_scalar*max(psi_min-psi,-10._r8))


is min_ftc_scalar this a parameter value read from parameter file that can be user-defined? can we put 10 MPa as a parameter for future testing?

When psi gets to about 2 MPA from psi_min, the weighting factor gets incredibly small, the 10 MPa here is just to avoid numerical problems with that exponent function. We could test the min_ftc_scalar though.. I don't expect the choice of min_ftc_scalar to have any noticable impact on model output, but I could try a few different values to see if anything happens. We have the utility parameter fates_dev_arbitrary that could aid in this.

xuchongang · 2024-04-26T19:14:07Z

biogeophys/FatesHydroWTFMod.F90

+          ! low suction
+          call get_min_ftc_weight(psi_min,psi,min_ftc_weight,dmin_ftc_weight_dpsi)
+
+          ftc = ftc*(1._r8 - min_ftc_weight) + min_ftc*min_ftc_weight


I really like this implementation and I think it should resolve a good chuck of issues of overshooting

xuchongang · 2024-04-26T19:16:09Z

biogeophys/FatesHydroWTFMod.F90

    else
-        dpsidth = -this%beta*this%psi_sat/this%th_sat * (th/this%th_sat)**(-this%beta-1._r8)
+       if(th<this%th_min) then


Good to check on the edge cases

xuchongang · 2024-04-26T19:17:17Z

biogeophys/FatesHydroWTFMod.F90

+    real(r8) :: pc
+    real(r8) :: kr
+    real(r8) :: dkr_dP
+    real(r8) :: sat_res
+    real(r8) :: alpha
+    real(r8) :: lambda
+    real(r8) :: Se
+    real(r8) :: deltaPc
+    real(r8) :: dSe_dpc
+    real(r8) :: dkr_dSe


I agree that it would be great to add comments on the units and meaning of each variables

xuchongang · 2024-04-26T19:19:43Z

functional_unit_testing/hydro/HydroUTestDriver.py

+
+    return(om_watsat,om_sucsat,om_bsw)
+
+def CCHParmsCosby84T5(zsoi,om_frac,sand_frac,clay_frac):


put some citation would be great

xuchongang · 2024-04-26T20:00:52Z

biogeophys/FatesPlantHydraulicsMod.F90

@@ -2767,6 +2784,20 @@ subroutine hydraulics_bc ( nsites, sites, bc_in, bc_out, dtime)
           end do

           ! Second pass, apply normalized weighting factors for fluxes
+


Good catch for low water content. If the sum of weight is close to zero, then the layer-specific water uptake is proportion of the root x water content here. It might be better to explain more on the new disaggregation

I updated the comment a little, should be more accurate. Does this look good @xuchongang ?

…ro, to enable first time-step issues in ctsm

rgknox · 2024-05-01T22:03:24Z

tests ok, tests_0501-150922de
note: I also tested without the protections added in the photosynthesis routine for small leaf areas, when I disabled that there was b4b on all tests.
note: the two hydro tests in the fates suite now pass (previously fail)

rgknox added 2 commits February 22, 2024 16:04

Added protections to Campbell Clapp-Hornberger water transfer functio…

2e8608c

…ns so that it will tolerate incredibly small water contents

reverting the cap on the smoothed cch function, it already has a prot…

f8dc237

…ection

glemieux self-requested a review February 26, 2024 18:01

glemieux approved these changes Mar 6, 2024

View reviewed changes

mpaiao reviewed Mar 7, 2024

View reviewed changes

rgknox added 2 commits March 13, 2024 13:53

Merge branch 'main' into cch-safemath

271282d

Updated error reporting in 1d taylor solve for hydro

fce4287

glemieux mentioned this pull request Mar 16, 2024

Add a recruitment carbon flux history variable #1153

Merged

4 tasks

correct minor build errors

13c2caa

glemieux mentioned this pull request Mar 19, 2024

prevent FATES-Hydro from failing upon restart with a drought deciduous PFT #1156

Merged

4 tasks

glemieux added the stuck Pull request is currently waiting on resolution of an issue that came up during review or testing label Mar 19, 2024

rgknox added PR status: Not Ready The author is signaling that this PR is a work in progress and not ready for integration. and removed stuck Pull request is currently waiting on resolution of an issue that came up during review or testing labels Mar 25, 2024

rgknox and others added 7 commits March 25, 2024 13:05

Adding ftc protections (ie 0 flux) for perilously low wcs

8f85de8

hydro safemath, looking into pinning zero ftc to min psi

1abba2f

added hydro code to force zero conductance a psi_min

35b396d

Getting zero flux at psi_min in fates hydro

57b10d3

adjusted ordering of initializing bounds on cch functions to avoid un…

a269473

…defined variables, reduce minimum suction of cch and fixed an error check

Merge tag 'sci.1.72.6_api.34.0.0' into cch-safemath-zeroflux

3a2b877

Bug fix. Adrianna Foster identified that trunks should be removed from the patch-level variable sum_fuel before being used to calculate rate of spread and fuel consumption.

Added btran to hydro restart, and protecting downscaling rhiz layer f…

cf05475

…luxes to soil layer fluxes when no conductance (trivial bug fix).

rgknox removed the PR status: Not Ready The author is signaling that this PR is a work in progress and not ready for integration. label Apr 17, 2024

jennykowalcz mentioned this pull request Apr 22, 2024

FATES-Hydro fails in very dry regions #1150

Closed

jennykowalcz mentioned this pull request Apr 22, 2024

FATES-Hydro: NaN/Inf error for soil matric potential #1154

Closed

jennykowalcz mentioned this pull request Apr 23, 2024

FATES-Hydro fails upon restart when using alternate soil water transfer functions #1157

Closed

bchristo reviewed Apr 23, 2024

View reviewed changes

xuchongang reviewed Apr 26, 2024

View reviewed changes

rgknox added 2 commits April 29, 2024 11:23

Updated some comments in low water potential protections

25fb73a

update to comments in hydro flux disaggregation code

a5d9fd9

disaggregating fluxes based on root length only when sumweights is ze…

ec27207

…ro, to enable first time-step issues in ctsm

rgknox merged commit e262198 into NGEET:main May 1, 2024
1 check was pending

glemieux added a commit to samsrabin/CTSM that referenced this pull request Jun 4, 2024

remove hydro tests that are now passing per NGEET/fates#1164

aed9d31

		@@ -44,6 +44,8 @@ module FatesHydroWTFMod
		! elastic-caviation region


		real(r8), parameter :: min_psi_cch = -9._r8 ! Minimum psi we are willing to track in cch


		return(om_watsat,om_sucsat,om_bsw)

		def CCHParmsCosby84T5(zsoi,om_frac,sand_frac,clay_frac):

		@@ -2767,6 +2784,20 @@ subroutine hydraulics_bc ( nsites, sites, bc_in, bc_out, dtime)
		end do

		! Second pass, apply normalized weighting factors for fluxes

Small number protections to CCH water transfer functions #1164

Small number protections to CCH water transfer functions #1164

Conversation

rgknox commented Feb 22, 2024 • edited Loading

Description:

Collaborators:

Expectation of Answer Changes:

Checklist

Documentation

Test Results:

rgknox commented Feb 26, 2024

jennykowalcz commented Feb 26, 2024

glemieux commented Mar 6, 2024 • edited Loading

jennykowalcz commented Mar 7, 2024

glemieux commented Mar 7, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rgknox Apr 29, 2024 • edited Loading

Choose a reason for hiding this comment

rgknox commented Apr 17, 2024

rgknox commented Apr 17, 2024

jennykowalcz commented Apr 17, 2024

rgknox commented Apr 17, 2024

jennykowalcz commented Apr 22, 2024

bchristo commented Apr 22, 2024 via email

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xuchongang left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rgknox commented May 1, 2024 • edited Loading

rgknox commented Feb 22, 2024 •

edited

Loading

glemieux commented Mar 6, 2024 •

edited

Loading

glemieux commented Mar 7, 2024 •

edited

Loading

rgknox Apr 29, 2024 •

edited

Loading

rgknox commented May 1, 2024 •

edited

Loading