Skip to content

Commit

Permalink
Update Validation vignette
Browse files Browse the repository at this point in the history
  • Loading branch information
@aidanmorales
aidanmorales committed Nov 18, 2024
1 parent c907125 commit 72606a8
Show file tree
Hide file tree
Showing 3 changed files with 189 additions and 4 deletions.
4 changes: 4 additions & 0 deletions NEWS.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,10 @@ Fixed a bug in `plot_qsm()` where color vectors or a vector element would throw

Random colors are now consistently brighter.

Update Validation vignette with more data.

Polish up documentation.

# rTwig 1.2.0

## New Features
Expand Down
83 changes: 83 additions & 0 deletions inst/extdata/validation2.csv
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
Dataset,SpCode,Tree,scientific.name,version,dbh.cm,ht.m,bdensity,Mt.TLS,Mt.DS,tdiff,tperr,colors
Burt et al. (2021),CAXH,1,Inga alba (Sw.) Willd.,Real Twig (TreeQSM v2.4.1),64.7,29.8,0.567629724,3520.351842,3960.074642,-439.7227996,-11.1039018,#CC79A7
Burt et al. (2021),CAXH,2,Hymenaea courbaril L.,Real Twig (TreeQSM v2.4.1),117.9,46.2,0.768870301,18119.27301,18584.21071,-464.9377017,-2.501788798,#CC79A7
Burt et al. (2021),CAXH,3,Tachigali paniculata var. alba (Duke) Dwyer,Real Twig (TreeQSM v2.4.1),90.5,34.9,0.638983869,8016.458444,8392.561074,-376.1026304,-4.481380917,#CC79A7
Burt et al. (2021),CAXH,4,Trattinnickia burserifolia Mart.,Real Twig (TreeQSM v2.4.1),69.7,35.2,0.566995516,5083.175244,5521.09504,-437.9197961,-7.931756163,#CC79A7
Burt et al. (2021),CAXH,1,Inga alba (Sw.) Willd.,TreeQSM v2.4.1,64.7,29.8,0.567629724,4549.256621,3960.074642,589.1819789,14.87805237,#999999
Burt et al. (2021),CAXH,2,Hymenaea courbaril L.,TreeQSM v2.4.1,117.9,46.2,0.768870301,20351.90849,18584.21071,1767.697779,9.511825963,#999999
Burt et al. (2021),CAXH,3,Tachigali paniculata var. alba (Duke) Dwyer,TreeQSM v2.4.1,90.5,34.9,0.638983869,10554.25085,8392.561074,2161.689777,25.75721235,#999999
Burt et al. (2021),CAXH,4,Trattinnickia burserifolia Mart.,TreeQSM v2.4.1,69.7,35.2,0.566995516,6752.746762,5521.09504,1231.651722,22.30810579,#999999
Demol et al. (2021a),FEXC,1,Fraxinus excelsior,Real Twig (TreeQSM v2.4.1),27.4,21.7,0.841436499,654.8083628,705.5,-50.69163719,-7.185207256,#CC79A7
Demol et al. (2021a),FEXC,2,Fraxinus excelsior,Real Twig (TreeQSM v2.4.1),22.6,20.6,0.838876526,392.5699292,416.101,-23.53107082,-5.655134407,#CC79A7
Demol et al. (2021a),FEXC,3,Fraxinus excelsior,Real Twig (TreeQSM v2.4.1),19.4,18.4,0.805016774,231.2264682,239.698,-8.471531816,-3.534252191,#CC79A7
Demol et al. (2021a),FEXC,4,Fraxinus excelsior,Real Twig (TreeQSM v2.4.1),20.4,21,0.761437381,262.8487591,266.02,-3.171240903,-1.192106196,#CC79A7
Demol et al. (2021a),FEXC,5,Fraxinus excelsior,Real Twig (TreeQSM v2.4.1),11.5,12.6,0.813753371,59.23764705,61.946,-2.708352948,-4.372119181,#CC79A7
Demol et al. (2021a),FEXC,6,Fraxinus excelsior,Real Twig (TreeQSM v2.4.1),30.9,18.3,0.815031118,550.0391014,543,7.039101414,1.296335435,#CC79A7
Demol et al. (2021a),FEXC,7,Fraxinus excelsior,Real Twig (TreeQSM v2.4.1),12.4,13.2,0.808442734,74.61749519,77.055,-2.437504809,-3.163331139,#CC79A7
Demol et al. (2021a),FEXC,8,Fraxinus excelsior,Real Twig (TreeQSM v2.4.1),28.6,21.5,0.796978502,728.7179329,789.621,-60.90306711,-7.712949264,#CC79A7
Demol et al. (2021a),FEXC,9,Fraxinus excelsior,Real Twig (TreeQSM v2.4.1),19.7,21.1,0.85131588,308.2053052,300.093,8.112305221,2.703263729,#CC79A7
Demol et al. (2021a),FEXC,10,Fraxinus excelsior,Real Twig (TreeQSM v2.4.1),12.6,13,0.76016082,69.13671109,61.5,7.636711093,12.41741641,#CC79A7
Demol et al. (2021a),FEXC,11,Fraxinus excelsior,Real Twig (TreeQSM v2.4.1),25.8,20.3,0.890391625,494.2796115,474,20.27961151,4.278399052,#CC79A7
Demol et al. (2021a),FEXC,13,Fraxinus excelsior,Real Twig (TreeQSM v2.4.1),35.3,19,0.906688723,922.0598452,904,18.05984519,1.997770486,#CC79A7
Demol et al. (2021a),FEXC,14,Fraxinus excelsior,Real Twig (TreeQSM v2.4.1),39.8,22.5,0.895750138,1514.605496,1444,70.60549558,4.889577256,#CC79A7
Demol et al. (2021a),FEXC,15,Fraxinus excelsior,Real Twig (TreeQSM v2.4.1),33.7,20.1,0.824422923,955.500389,905.872,49.62838903,5.478521141,#CC79A7
Demol et al. (2021a),FEXC,16,Fraxinus excelsior,Real Twig (TreeQSM v2.4.1),22.6,18.2,0.800253433,388.6520542,406,-17.34794581,-4.272893058,#CC79A7
Demol et al. (2021a),FEXC,1,Fraxinus excelsior,TreeQSM v2.4.1,27.4,21.7,0.841436499,958.8642426,705.5,253.3642426,35.91272042,#999999
Demol et al. (2021a),FEXC,2,Fraxinus excelsior,TreeQSM v2.4.1,22.6,20.6,0.838876526,597.6495983,416.101,181.5485983,43.63089689,#999999
Demol et al. (2021a),FEXC,3,Fraxinus excelsior,TreeQSM v2.4.1,19.4,18.4,0.805016774,339.7795276,239.698,100.0815276,41.75317593,#999999
Demol et al. (2021a),FEXC,4,Fraxinus excelsior,TreeQSM v2.4.1,20.4,21,0.761437381,564.5718611,266.02,298.5518611,112.2291035,#999999
Demol et al. (2021a),FEXC,5,Fraxinus excelsior,TreeQSM v2.4.1,11.5,12.6,0.813753371,79.44182293,61.946,17.49582293,28.24366857,#999999
Demol et al. (2021a),FEXC,6,Fraxinus excelsior,TreeQSM v2.4.1,30.9,18.3,0.815031118,622.4378698,543,79.43786978,14.62944195,#999999
Demol et al. (2021a),FEXC,7,Fraxinus excelsior,TreeQSM v2.4.1,12.4,13.2,0.808442734,118.8239297,77.055,41.76892972,54.20664424,#999999
Demol et al. (2021a),FEXC,8,Fraxinus excelsior,TreeQSM v2.4.1,28.6,21.5,0.796978502,2039.574951,789.621,1249.953951,158.2979621,#999999
Demol et al. (2021a),FEXC,9,Fraxinus excelsior,TreeQSM v2.4.1,19.7,21.1,0.85131588,632.7164469,300.093,332.6234469,110.8401219,#999999
Demol et al. (2021a),FEXC,10,Fraxinus excelsior,TreeQSM v2.4.1,12.6,13,0.76016082,103.9358129,61.5,42.43581294,69.00132185,#999999
Demol et al. (2021a),FEXC,11,Fraxinus excelsior,TreeQSM v2.4.1,25.8,20.3,0.890391625,721.1856137,474,247.1856137,52.14886365,#999999
Demol et al. (2021a),FEXC,13,Fraxinus excelsior,TreeQSM v2.4.1,35.3,19,0.906688723,1153.432131,904,249.432131,27.59204988,#999999
Demol et al. (2021a),FEXC,14,Fraxinus excelsior,TreeQSM v2.4.1,39.8,22.5,0.895750138,2947.736367,1444,1503.736367,104.1368675,#999999
Demol et al. (2021a),FEXC,15,Fraxinus excelsior,TreeQSM v2.4.1,33.7,20.1,0.824422923,1302.704604,905.872,396.8326043,43.80669723,#999999
Demol et al. (2021a),FEXC,16,Fraxinus excelsior,TreeQSM v2.4.1,22.6,18.2,0.800253433,473.8934424,406,67.89344243,16.72252277,#999999
Hackenberg et al. (2015b),QUPE,1,Quercus petraea,Real Twig (TreeQSM v2.4.1),24.5,25.5,0.53,383.3898158,458.82,-75.43018425,-16.44003841,#CC79A7
Hackenberg et al. (2015b),QUPE,2,Quercus petraea,Real Twig (TreeQSM v2.4.1),24.2,26.6,0.52,332.0954423,311.03,21.06544231,6.772800794,#CC79A7
Hackenberg et al. (2015b),QUPE,3,Quercus petraea,Real Twig (TreeQSM v2.4.1),31.7,29,0.52,617.1553937,595.32,21.8353937,3.667841446,#CC79A7
Hackenberg et al. (2015b),QUPE,4,Quercus petraea,Real Twig (TreeQSM v2.4.1),26.3,28.6,0.54,459.0364945,517.56,-58.52350545,-11.30757892,#CC79A7
Hackenberg et al. (2015b),QUPE,5,Quercus petraea,Real Twig (TreeQSM v2.4.1),28.7,24.8,0.53,463.5387013,412.24,51.29870129,12.44389222,#CC79A7
Hackenberg et al. (2015b),QUPE,6,Quercus petraea,Real Twig (TreeQSM v2.4.1),27.5,29.4,0.51,428.9926815,468.74,-39.74731851,-8.479608848,#CC79A7
Hackenberg et al. (2015b),QUPE,7,Quercus petraea,Real Twig (TreeQSM v2.4.1),24,27.2,0.52,345.3110327,366.09,-20.77896735,-5.675917766,#CC79A7
Hackenberg et al. (2015b),QUPE,8,Quercus petraea,Real Twig (TreeQSM v2.4.1),30,27.6,0.51,566.9824206,632.91,-65.92757941,-10.41658046,#CC79A7
Hackenberg et al. (2015b),QUPE,9,Quercus petraea,Real Twig (TreeQSM v2.4.1),29.2,31.1,0.52,582.0332004,581.76,0.27320041,0.046961017,#CC79A7
Hackenberg et al. (2015b),QUPE,10,Quercus petraea,Real Twig (TreeQSM v2.4.1),27.1,22.7,0.52,454.0879487,411.82,42.26794867,10.26369498,#CC79A7
Hackenberg et al. (2015b),QUPE,11,Quercus petraea,Real Twig (TreeQSM v2.4.1),25.7,25.8,0.51,418.5986318,452.16,-33.56136825,-7.422454054,#CC79A7
Hackenberg et al. (2015b),QUPE,12,Quercus petraea,Real Twig (TreeQSM v2.4.1),29.4,27.3,0.53,634.5588988,589.56,44.99889882,7.632624129,#CC79A7
Hackenberg et al. (2015b),QUPE,1,Quercus petraea,TreeQSM v2.4.1,24.5,25.5,0.53,730.1677929,458.82,271.3477929,59.14035851,#999999
Hackenberg et al. (2015b),QUPE,2,Quercus petraea,TreeQSM v2.4.1,24.2,26.6,0.52,505.6675206,311.03,194.6375206,62.57837528,#999999
Hackenberg et al. (2015b),QUPE,3,Quercus petraea,TreeQSM v2.4.1,31.7,29,0.52,1032.735867,595.32,437.4158668,73.47575535,#999999
Hackenberg et al. (2015b),QUPE,4,Quercus petraea,TreeQSM v2.4.1,26.3,28.6,0.54,682.8817439,517.56,165.3217439,31.94252722,#999999
Hackenberg et al. (2015b),QUPE,5,Quercus petraea,TreeQSM v2.4.1,28.7,24.8,0.53,783.5947723,412.24,371.3547723,90.08217841,#999999
Hackenberg et al. (2015b),QUPE,6,Quercus petraea,TreeQSM v2.4.1,27.5,29.4,0.51,770.5278421,468.74,301.7878421,64.3827798,#999999
Hackenberg et al. (2015b),QUPE,7,Quercus petraea,TreeQSM v2.4.1,24,27.2,0.52,679.1220275,366.09,313.0320275,85.50685008,#999999
Hackenberg et al. (2015b),QUPE,8,Quercus petraea,TreeQSM v2.4.1,30,27.6,0.51,1233.787502,632.91,600.8775018,94.93885414,#999999
Hackenberg et al. (2015b),QUPE,9,Quercus petraea,TreeQSM v2.4.1,29.2,31.1,0.52,916.3503323,581.76,334.5903323,57.51346472,#999999
Hackenberg et al. (2015b),QUPE,10,Quercus petraea,TreeQSM v2.4.1,27.1,22.7,0.52,1034.382772,411.82,622.5627718,151.1735156,#999999
Hackenberg et al. (2015b),QUPE,11,Quercus petraea,TreeQSM v2.4.1,25.7,25.8,0.51,745.5544883,452.16,293.3944883,64.88731606,#999999
Hackenberg et al. (2015b),QUPE,12,Quercus petraea,TreeQSM v2.4.1,29.4,27.3,0.53,951.6331443,589.56,362.0731443,61.4141299,#999999
Harvard Forest (2017),RM,1,Acer rubrum,Real Twig (TreeQSM v2.4.1),28.7,22.7,0.496704781,376.7341468,382.954173,-6.220026243,-1.624222082,#CC79A7
Harvard Forest (2017),RM,105,Acer rubrum,Real Twig (TreeQSM v2.4.1),7.6,11,0.535881614,20.16884315,17.26495136,2.903891796,16.81957705,#CC79A7
Harvard Forest (2017),RM,126,Acer rubrum,Real Twig (TreeQSM v2.4.1),21.8,23.1,0.465509259,202.6337802,243.1451421,-40.51136194,-16.66139064,#CC79A7
Harvard Forest (2017),RM,133,Acer rubrum,Real Twig (TreeQSM v2.4.1),11.9,13.4,0.576742502,61.45724226,57.5517939,3.905448365,6.785971559,#CC79A7
Harvard Forest (2017),RM,181,Acer rubrum,Real Twig (TreeQSM v2.4.1),10.7,16.9,0.570395849,57.84103821,54.91882797,2.922210239,5.320962494,#CC79A7
Harvard Forest (2017),RM,1,Acer rubrum,TreeQSM v2.4.1,28.7,22.7,0.496704781,575.6184105,382.954173,192.6642375,50.30999818,#999999
Harvard Forest (2017),RM,105,Acer rubrum,TreeQSM v2.4.1,7.6,11,0.535881614,95.15351077,17.26495136,77.88855941,451.1368598,#999999
Harvard Forest (2017),RM,126,Acer rubrum,TreeQSM v2.4.1,21.8,23.1,0.465509259,326.7478576,243.1451421,83.60271544,34.38387241,#999999
Harvard Forest (2017),RM,133,Acer rubrum,TreeQSM v2.4.1,11.9,13.4,0.576742502,208.3535554,57.5517939,150.8017615,262.027908,#999999
Harvard Forest (2017),RM,181,Acer rubrum,TreeQSM v2.4.1,10.7,16.9,0.570395849,155.004495,54.91882797,100.085667,182.2429041,#999999
Harvard Forest (2017),RO,1,Quercus rubra,Real Twig (TreeQSM v2.4.1),36.3,21.6,0.491856353,757.9426731,809.2078735,-51.26520044,-6.33523253,#CC79A7
Harvard Forest (2017),RO,36,Quercus rubra,Real Twig (TreeQSM v2.4.1),19.3,21.2,0.530650181,165.6249618,172.7666917,-7.141729938,-4.13374237,#CC79A7
Harvard Forest (2017),RO,75,Quercus rubra,Real Twig (TreeQSM v2.4.1),32.3,22.2,0.550764265,632.0010209,641.1450704,-9.144049523,-1.426206009,#CC79A7
Harvard Forest (2017),RO,234,Quercus rubra,Real Twig (TreeQSM v2.4.1),26.7,23.5,0.543130536,391.2388592,400.5062915,-9.267432296,-2.313929267,#CC79A7
Harvard Forest (2017),RO,251,Quercus rubra,Real Twig (TreeQSM v2.4.1),50.3,24.1,0.551479523,1470.513384,1417.312108,53.20127551,3.753673957,#CC79A7
Harvard Forest (2017),RO,1,Quercus rubra,TreeQSM v2.4.1,36.3,21.6,0.491856353,1018.309568,809.2078735,209.1016949,25.84029417,#999999
Harvard Forest (2017),RO,36,Quercus rubra,TreeQSM v2.4.1,19.3,21.2,0.530650181,251.0194896,172.7666917,78.25279783,45.29391461,#999999
Harvard Forest (2017),RO,75,Quercus rubra,TreeQSM v2.4.1,32.3,22.2,0.550764265,1252.436867,641.1450704,611.2917969,95.34375684,#999999
Harvard Forest (2017),RO,234,Quercus rubra,TreeQSM v2.4.1,26.7,23.5,0.543130536,676.0784211,400.5062915,275.5721296,68.80594274,#999999
Harvard Forest (2017),RO,251,Quercus rubra,TreeQSM v2.4.1,50.3,24.1,0.551479523,2110.267222,1417.312108,692.9551134,48.89220301,#999999
106 changes: 102 additions & 4 deletions vignettes/Validation.Rmd
View file
Original file line number Diff line number Diff line change
Expand Up @@ -25,12 +25,110 @@ library(yardstick)
library(gt)
```

How do we know if Real Twig returns accurate volume estimates? We rigorously tested our method against multiple high quality reference data sets that were both laser scanner and destructively sampled. The laser scanning was done in leaf-off conditions with a Riegl VZ-400. We used different versions of TreeQSM with the same input parameters per tree and different data sets to test our method. A detailed discussion of the results and implications can be found in *Morales & MacFarlane (2024)*: <https://doi.org/10.1093/forestry/cpae046>

How do we know if Real Twig returns accurate volume estimates? We rigorously tested our method against a high quality reference data set that were both laser scanner and destructively sampled. The laser scanning was done in leaf-off conditions with a Riegl VZ-400. The destructive sampling contains total branch and main stem dry mass, and also basic density for both the main stem and the branches. We used different versions of TreeQSM with the same input parameters per tree to test our model.
The graph and table below contain destructive sampling with total mass and basic density (wood + bark) across four data sets, three of which are publicly available and are linked below. The mass estimates and statistics use TreeQSM v2.4.1 with its built in tapering compared to Real Twig on the same QSMs.

- Burt et al. (2021): <https://zenodo.org/records/4056899>, <https://zenodo.org/records/4056903>

- Demol et al. (2021): <https://zenodo.org/records/4557401>

- Hackenberg et al. (2015b): <https://www.simpleforest.org/pages/dataQuercusPetraea.html>

```{r, echo=FALSE, results='hide', message=FALSE, warning=FALSE}
# Import all data
file <- system.file("extdata/validation2.csv", package = "rTwig")
data_comp <- read.csv(file)
p_all <- ggplot(data = data_comp, aes(x = dbh.cm, y = tperr, color = version, fill = version)) +
geom_point(aes(shape = Dataset, color = version, fill = version)) +
geom_smooth(method = "gam", formula = y ~ s(x, bs = "cs")) +
geom_hline(yintercept = 0) +
labs(
x = "DBH (cm)",
y = "Total Mass Error (%)",
color = "Method",
fill = "Method",
shape = "Dataset"
) +
theme_classic() +
scale_color_manual(values = c("#D41159", "#1A85FF")) +
scale_fill_manual(values = c("#D41159", "#1A85FF")) +
geom_hline(
yintercept = 10,
linetype = "dashed",
color = "black",
linewidth = 0.25
) +
geom_hline(
yintercept = -10,
linetype = "dashed",
color = "black",
linewidth = 0.25,
show.legend = TRUE
)
```

```{r, echo=FALSE, fig.width=7, fig.height=4, fig.align='center', warning=FALSE}
p_all
```

```{r, echo=FALSE, warning=FALSE}
### ALL TREES ##################################################################
total_stats <- data_comp %>%
ungroup() %>%
group_by(version) %>%
summarize(
MRE.pct = mean(tperr, na.rm = TRUE),
RMSE.kg = sqrt(mean(tdiff^2, na.rm = TRUE)),
RRMSE.pct = RMSE.kg / mean(Mt.DS, na.rm = TRUE) * 100
)
CCC_total <- data_comp %>%
ungroup() %>%
group_by(version) %>%
yardstick::ccc(Mt.TLS, Mt.DS) %>%
select(.estimate) %>%
rename(CCC = 1)
total_stats <- bind_cols(total_stats, CCC_total) %>%
mutate(type = "total") %>%
relocate(type, .before = MRE.pct) %>%
select(-type)
data <- total_stats %>%
mutate_if(is.numeric, round, 3) %>%
rename(
"Mean Relative Error (%)" = MRE.pct,
"RMSE (kg)" = RMSE.kg,
"Relative RMSE (%)" = RRMSE.pct
)
data %>%
gt() %>%
# tab_header(title = unique(.$`_data`$version)) %>%
# cols_hide(version) %>%
cols_label(
version = "Method",
) %>%
cols_align(
align = "center",
columns = everything()
) %>%
tab_options(
table.border.top.style = "hidden",
table.border.bottom.style = "hidden",
table_body.hlines.color = "white",
table.font.color = "black",
heading.border.bottom.color = "black",
column_labels.border.bottom.color = "black",
table_body.border.bottom.color = "black"
)
```

## TreeQSM v2.4.1

Below are the mass estimates and statistics using TreeQSM v2.4.1 with its built in tapering and Real Twig applied to the same QSMs.
The following figures look at the Harvard Forest data set using different versions of TreeQSM. The destructive sampling data contains total branch and main stem dry mass, and also basic density for both the main stem and the branches. This allows us to test for compensating errors, and look at the true differences between TreeQSM versions. We used the same input parameters per tree and TreeQSM version to test our model.

```{r, echo=FALSE, results='hide', message=FALSE, warning=FALSE}
# Import Data
Expand Down Expand Up @@ -552,9 +650,9 @@ data[[1]] %>%

## SimpleForest

It is important to note that Real Twig was not tested with SimpleForest during its development. While Real Twig does improve volume estimates for SimpleForest versus its built in allometric corrections, there are still improvements to be made, as SimpleForest QSM cylinders are generally much more overestimated than TreeQSM cylinders, making the identification of "good" cylinders difficult.
Real Twig was not tested with SimpleForest during its development. While Real Twig does improve volume estimates for SimpleForest versus its built in allometric corrections, there are still improvements to be made, as SimpleForest QSM cylinders are generally much more overestimated than TreeQSM cylinders, making the identification of "good" cylinders difficult.

Below are the mass estimates and statistics using SimpleForest v5.3.2 with its built in vessel volume correction, and Real Twig applied to the same QSMs.
Below are the mass estimates and statistics using SimpleForest v5.3.2 with its built in vessel volume correction, and Real Twig applied to the same QSMs.

```{r, echo=FALSE, results='hide', message=FALSE, warning=FALSE}
# Import Data
Expand Down

0 comments on commit 72606a8

Please sign in to comment.