Model Assessment

Computes various metrics for assessing the quality of a fitted flood frequency model. Metrics include accuracy (residual statistics), fitting efficiency (information criteria), and uncertainty (coverage based metrics using confidence intervals).

For NS-FFA: The metrics are computed from the normalized empirical/theoretical quantiles, which are determined based on the selected distribution family. Therefore, metrics from stationary and nonstationary models should not be compared.

Usage

model_assessment(
  data,
  distribution,
  method,
  prior = NULL,
  ns_years = NULL,
  ns_structure = NULL,
  alpha = 0.05,
  pp_formula = "Weibull",
  ci = NULL
)

Arguments

data

Numeric vector of observed annual maximum series values. Must be strictly positive, finite, and not missing.

distribution

A three-character code indicating the distribution family. Must be "GUM", "NOR", "LNO", "GEV", "GLO", "GNO", "PE3", "LP3", or "WEI".

method

Character scalar specifying the estimation method. Must be "L-moments", "MLE", or "GMLE".

prior

Numeric vector of length 2. Specifies the parameters of the Beta prior for the shape parameter \(\kappa\).

ns_years

For NS-FFA only: Numeric vector of observation years corresponding to data. Must be the same length as data and strictly increasing.

ns_structure

For NS-FFA only: Named list indicating which distribution parameters are modeled as nonstationary. Must contain two logical scalars:

location: If TRUE, the location parameter has a linear temporal trend.
scale: If TRUE, the scale parameter has a linear temporal trend.

alpha

Numeric scalar in \([0.01, 0.1]\). The significance level for confidence intervals or hypothesis tests. Default is 0.05.

pp_formula

Character string specifying the plotting position formula. Must "Weibull" (default), "Blom", "Cunnane", "Gringorten", or "Hazen".

ci

For S-FFA only. Dataframe containing return periods (in the column periods) and confidence intervals (in the columns lower and upper). Dataframes in this format can be generated with uncertainty_bootstrap(), uncertainty_rfpl(), or uncertainty_rfgpl().

Value

List containing the results of model assessment:

data: The data argument.
q_theoretical: The theoretical return level estimates based on the plotting positions. Normalized for nonstationary models.
q_empirical: The empirical return levels. Normalized for nonstationary models.
metrics: A list of model assessment metrics (see details).

Details

These metrics are are computed for all models:

AIC_MLL: Akaike Information Criterion, computed using the maximum log-likelihood.
BIC_MLL: Bayesian Information Criterion, computed using the maximum log-likelihood.
R2: Coefficient of determination from linear regression of estimates vs. data.
RMSE: Root mean squared error of quantile estimates.
bias: Mean bias of quantile estimates.
AIC_RMSE: Akaike Information Criterion, computed using the RMSE.
BIC_RMSE: Bayesian Information Criterion, computed using the RMSE.

These metrics are only computed if the ci argument is provided:

AW: Average width of the confidence interval(s).
POC: Percent of observations covered by the confidence interval(s).
CWI: Confidence width index, a metric that combines AW and POC.

Examples

# Initialize example data and params
data <- rnorm(n = 100, mean = 100, sd = 10)
params <- c(100, 10)

# Perform uncertainty analysis
ci <- uncertainty_bootstrap(data, "NOR", "L-moments")$ci

# Run model assessment
model_assessment(data, "NOR", "L-moments", ci = ci)
#> $data
#>   [1]  75.23320  86.10714 108.21163  99.17586  93.73108  89.23005  93.14664
#>   [8] 103.55095 109.96582 109.94133  89.48270 100.77882 105.86229  96.05944
#>  [15] 105.74290 101.84364  88.81079  88.19487 104.83221 107.28678  90.25614
#>  [22]  85.70304  87.53543  75.85501  81.29131 112.34168 102.36275  97.20200
#>  [29]  95.89982 114.17581 106.12786  80.78296 101.99865  90.26977  68.72037
#>  [36]  96.84019  94.40152  94.10852  90.60605 104.78407  91.14503 101.86453
#>  [43]  88.78454  95.55080  98.17331 100.36910 102.09306  98.87816  96.74392
#>  [50]  75.56170 101.86518 117.44204  99.59156  98.69751  95.69942  88.03281
#>  [57]  98.55958 110.30797  96.95703 106.25070 101.96883 110.42065  93.68646
#>  [64] 103.34383  94.88873  94.56708  86.10628 110.11812 101.27535 101.56512
#>  [71] 103.04107 111.97045 102.72246  91.78099 103.24926 107.68759 105.03399
#>  [78]  83.06022 111.48481 101.06242 104.15816 104.91312 106.74244 103.73173
#>  [85] 112.11088 106.51197 117.53953 100.23531 105.69572  94.61341 101.11590
#>  [92]  90.87626 100.45123 108.96800  87.22664 111.80982  85.37089 100.92801
#>  [99] 100.60390  90.05612
#> 
#> $q_theoretical
#>   [1] 120.77319 118.16367 116.50837 115.26338 114.25084 113.38913 112.63368
#>   [8] 111.95736 111.34236 110.77631 110.25030 109.75765 109.29320 108.85293
#>  [15] 108.43360 108.03258 107.64768 107.27709 106.91927 106.57288 106.23681
#>  [22] 105.91006 105.59178 105.28119 104.97762 104.68048 104.38922 104.10336
#>  [29] 103.82245 103.54608 103.27391 103.00557 102.74077 102.47922 102.22064
#>  [36] 101.96478 101.71142 101.46032 101.21127 100.96409 100.71858 100.47455
#>  [43] 100.23184  99.99027  99.74969  99.50993  99.27084  99.03226  98.79406
#>  [50]  98.55607  98.31816  98.08018  97.84197  97.60340  97.36431  97.12455
#>  [57]  96.88396  96.64240  96.39968  96.15566  95.91014  95.66296  95.41392
#>  [64]  95.16282  94.90946  94.65360  94.39502  94.13346  93.86866  93.60033
#>  [71]  93.32815  93.05179  92.77088  92.48501  92.19375  91.89661  91.59305
#>  [78]  91.28246  90.96417  90.63743  90.30135  89.95497  89.59714  89.22655
#>  [85]  88.84166  88.44063  88.02131  87.58104  87.11659  86.62393  86.09792
#>  [92]  85.53188  84.91687  84.24055  83.48511  82.62340  81.61085  80.36586
#>  [99]  78.71057  76.10104
#> 
#> $q_empirical
#>   [1] 117.53953 117.44204 114.17581 112.34168 112.11088 111.97045 111.80982
#>   [8] 111.48481 110.42065 110.30797 110.11812 109.96582 109.94133 108.96800
#>  [15] 108.21163 107.68759 107.28678 106.74244 106.51197 106.25070 106.12786
#>  [22] 105.86229 105.74290 105.69572 105.03399 104.91312 104.83221 104.78407
#>  [29] 104.15816 103.73173 103.55095 103.34383 103.24926 103.04107 102.72246
#>  [36] 102.36275 102.09306 101.99865 101.96883 101.86518 101.86453 101.84364
#>  [43] 101.56512 101.27535 101.11590 101.06242 100.92801 100.77882 100.60390
#>  [50] 100.45123 100.36910 100.23531  99.59156  99.17586  98.87816  98.69751
#>  [57]  98.55958  98.17331  97.20200  96.95703  96.84019  96.74392  96.05944
#>  [64]  95.89982  95.69942  95.55080  94.88873  94.61341  94.56708  94.40152
#>  [71]  94.10852  93.73108  93.68646  93.14664  91.78099  91.14503  90.87626
#>  [78]  90.60605  90.26977  90.25614  90.05612  89.48270  89.23005  88.81079
#>  [85]  88.78454  88.19487  88.03281  87.53543  87.22664  86.10714  86.10628
#>  [92]  85.70304  85.37089  83.06022  81.29131  80.78296  75.85501  75.56170
#>  [99]  75.23320  68.72037
#> 
#> $metrics
#> $metrics$AIC_MLL
#> [1] 739.9498
#> 
#> $metrics$BIC_MLL
#> [1] 745.1601
#> 
#> $metrics$R2
#> [1] 0.9755023
#> 
#> $metrics$RMSE
#> [1] 1.524231
#> 
#> $metrics$bias
#> [1] -2.842448e-15
#> 
#> $metrics$AIC_RMSE
#> [1] 46.14899
#> 
#> $metrics$BIC_RMSE
#> [1] 51.35933
#> 
#> $metrics$AW
#> [1] 4.441414
#> 
#> $metrics$POC
#> [1] 95.91837
#> 
#> $metrics$CWI
#> [1] 4.360582
#> 
#>

Usage

Arguments

Value

Details

See also

Examples