Model Assessment

Computes various metrics for assessing the quality of a fitted flood frequency model. Metrics include accuracy (residual statistics), fitting efficiency (information criteria), and uncertainty (coverage based metrics using confidence intervals).

For NS-FFA: The metrics are computed from the normalized empirical/theoretical quantiles, which are determined based on the selected distribution family. Therefore, metrics from stationary and nonstationary models should not be compared.

Usage

model_assessment(
  data,
  distribution,
  method,
  prior = NULL,
  ns_years = NULL,
  ns_structure = NULL,
  alpha = 0.05,
  pp_formula = "Weibull",
  ci = NULL
)

Arguments

data

Numeric vector of observed annual maximum series values. Must be strictly positive, finite, and not missing.

distribution

A three-character code indicating the distribution family. Must be "GUM", "NOR", "LNO", "GEV", "GLO", "GNO", "PE3", "LP3", or "WEI".

method

Character scalar specifying the estimation method. Must be "L-moments", "MLE", or "GMLE".

prior

Numeric vector of length 2. Specifies the parameters of the Beta prior for the shape parameter \(\kappa\).

ns_years

For NS-FFA only: Numeric vector of observation years corresponding to data. Must be the same length as data and strictly increasing.

ns_structure

For NS-FFA only: Named list indicating which distribution parameters are modeled as nonstationary. Must contain two logical scalars:

location: If TRUE, the location parameter has a linear temporal trend.
scale: If TRUE, the scale parameter has a linear temporal trend.

alpha

Numeric scalar in \([0.01, 0.1]\). The significance level for confidence intervals or hypothesis tests. Default is 0.05.

pp_formula

Character string specifying the plotting position formula. Must "Weibull" (default), "Blom", "Cunnane", "Gringorten", or "Hazen".

ci

For S-FFA only. Dataframe containing return periods (in the column periods) and confidence intervals (in the columns lower and upper). Dataframes in this format can be generated with uncertainty_bootstrap(), uncertainty_rfpl(), or uncertainty_rfgpl().

Value

List containing the results of model assessment:

data: The data argument.
q_theoretical: The theoretical return level estimates based on the plotting positions. Normalized for nonstationary models.
q_empirical: The empirical return levels. Normalized for nonstationary models.
metrics: A list of model assessment metrics (see details).

Details

These metrics are are computed for all models:

AIC_MLL: Akaike Information Criterion, computed using the maximum log-likelihood.
BIC_MLL: Bayesian Information Criterion, computed using the maximum log-likelihood.
R2: Coefficient of determination from linear regression of estimates vs. data.
RMSE: Root mean squared error of quantile estimates.
bias: Mean bias of quantile estimates.
AIC_RMSE: Akaike Information Criterion, computed using the RMSE.
BIC_RMSE: Bayesian Information Criterion, computed using the RMSE.

These metrics are only computed if the ci argument is provided:

AW: Average width of the confidence interval(s).
POC: Percent of observations covered by the confidence interval(s).
CWI: Confidence width index, a metric that combines AW and POC.

Examples

# Initialize example data and params
data <- rnorm(n = 100, mean = 100, sd = 10)
params <- c(100, 10)

# Perform uncertainty analysis
ci <- uncertainty_bootstrap(data, "NOR", "L-moments")$ci

# Run model assessment
model_assessment(data, "NOR", "L-moments", ci = ci)
#> $data
#>   [1] 108.11776  98.44905  91.99071  75.23320  86.10714 108.21163  99.17586
#>   [8]  93.73108  89.23005  93.14664 103.55095 109.96582 109.94133  89.48270
#>  [15] 100.77882 105.86229  96.05944 105.74290 101.84364  88.81079  88.19487
#>  [22] 104.83221 107.28678  90.25614  85.70304  87.53543  75.85501  81.29131
#>  [29] 112.34168 102.36275  97.20200  95.89982 114.17581 106.12786  80.78296
#>  [36] 101.99865  90.26977  68.72037  96.84019  94.40152  94.10852  90.60605
#>  [43] 104.78407  91.14503 101.86453  88.78454  95.55080  98.17331 100.36910
#>  [50] 102.09306  98.87816  96.74392  75.56170 101.86518 117.44204  99.59156
#>  [57]  98.69751  95.69942  88.03281  98.55958 110.30797  96.95703 106.25070
#>  [64] 101.96883 110.42065  93.68646 103.34383  94.88873  94.56708  86.10628
#>  [71] 110.11812 101.27535 101.56512 103.04107 111.97045 102.72246  91.78099
#>  [78] 103.24926 107.68759 105.03399  83.06022 111.48481 101.06242 104.15816
#>  [85] 104.91312 106.74244 103.73173 112.11088 106.51197 117.53953 100.23531
#>  [92] 105.69572  94.61341 101.11590  90.87626 100.45123 108.96800  87.22664
#>  [99] 111.80982  85.37089
#> 
#> $q_theoretical
#>   [1] 120.93439 118.31418 116.65210 115.40201 114.38531 113.52007 112.76153
#>   [8] 112.08244 111.46492 110.89656 110.36839 109.87371 109.40737 108.96529
#>  [15] 108.54425 108.14158 107.75511 107.38300 107.02371 106.67591 106.33846
#>  [22] 106.01037 105.69078 105.37892 105.07411 104.77575 104.48330 104.19626
#>  [29] 103.91420 103.63671 103.36341 103.09398 102.82810 102.56547 102.30583
#>  [36] 102.04893 101.79452 101.54240 101.29233 101.04414 100.79762 100.55259
#>  [43] 100.30889 100.06633  99.82476  99.58402  99.34395  99.10440  98.86522
#>  [50]  98.62626  98.38737  98.14841  97.90923  97.66968  97.42961  97.18887
#>  [57]  96.94729  96.70474  96.46103  96.21601  95.96949  95.72129  95.47123
#>  [64]  95.21910  94.96470  94.70780  94.44816  94.18553  93.91964  93.65021
#>  [71]  93.37692  93.09942  92.81736  92.53033  92.23787  91.93951  91.63471
#>  [78]  91.32285  91.00326  90.67517  90.33772  89.98992  89.63062  89.25852
#>  [85]  88.87204  88.46938  88.04833  87.60626  87.13991  86.64523  86.11707
#>  [92]  85.54871  84.93118  84.25209  83.49355  82.62831  81.61162  80.36153
#>  [99]  78.69945  76.07924
#> 
#> $q_empirical
#>   [1] 117.53953 117.44204 114.17581 112.34168 112.11088 111.97045 111.80982
#>   [8] 111.48481 110.42065 110.30797 110.11812 109.96582 109.94133 108.96800
#>  [15] 108.21163 108.11776 107.68759 107.28678 106.74244 106.51197 106.25070
#>  [22] 106.12786 105.86229 105.74290 105.69572 105.03399 104.91312 104.83221
#>  [29] 104.78407 104.15816 103.73173 103.55095 103.34383 103.24926 103.04107
#>  [36] 102.72246 102.36275 102.09306 101.99865 101.96883 101.86518 101.86453
#>  [43] 101.84364 101.56512 101.27535 101.11590 101.06242 100.77882 100.45123
#>  [50] 100.36910 100.23531  99.59156  99.17586  98.87816  98.69751  98.55958
#>  [57]  98.44905  98.17331  97.20200  96.95703  96.84019  96.74392  96.05944
#>  [64]  95.89982  95.69942  95.55080  94.88873  94.61341  94.56708  94.40152
#>  [71]  94.10852  93.73108  93.68646  93.14664  91.99071  91.78099  91.14503
#>  [78]  90.87626  90.60605  90.26977  90.25614  89.48270  89.23005  88.81079
#>  [85]  88.78454  88.19487  88.03281  87.53543  87.22664  86.10714  86.10628
#>  [92]  85.70304  85.37089  83.06022  81.29131  80.78296  75.85501  75.56170
#>  [99]  75.23320  68.72037
#> 
#> $metrics
#> $metrics$AIC_MLL
#> [1] 740.5313
#> 
#> $metrics$BIC_MLL
#> [1] 745.7416
#> 
#> $metrics$R2
#> [1] 0.976168
#> 
#> $metrics$RMSE
#> [1] 1.50706
#> 
#> $metrics$bias
#> [1] -3.552714e-15
#> 
#> $metrics$AIC_RMSE
#> [1] 45.01606
#> 
#> $metrics$BIC_RMSE
#> [1] 50.2264
#> 
#> $metrics$AW
#> [1] 4.459609
#> 
#> $metrics$POC
#> [1] 95.91837
#> 
#> $metrics$CWI
#> [1] 4.378445
#> 
#>

Usage

Arguments

Value

Details

See also

Examples