Calculate scoring statistics from predictions using scoringutils.

This performs a range of quantile based, and if cumulative distribution functions are available, continuous scoring metrics for each estimate time-point.

Usage

score_estimate(est, obs, lags = NULL)

Arguments

est

a dataframe of estimates of incidence, growth rate of reproduction number based off a simulation or data with known parameters.

obs

a dataframe of the ground truth, sharing the same grouping as est with at least one column(s) named XXX.obs with XXX being one of rt,growth or incidence or any other column group predicted in est.

lags

a data frame of estimate types and lags as output by quantify_lag() if multiple models are included then the columns must match those in obs.

Value

a dataframe of scoring metrics

Examples

tmp2 = test_bpm %>% sim_summarise_linelist()

withr::with_options(list("ggoutbreak.keep_cdf"=TRUE),{
   est = tmp2 %>% poisson_locfit_model() %>% rt_from_incidence()
})

obs = tmp2 %>% dplyr::mutate(rt.obs = rt.weighted)
score_estimate(est,obs) %>% dplyr::glimpse()
#> Rows: 159
#> Columns: 14
#> Groups: model, statistic, estimate [2]
#> $ statistic          <chr> "infections", "infections", "infections", "infectio…
#> $ time               <time_prd> 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, …
#> $ estimate           <chr> "rt", "rt", "rt", "rt", "rt", "rt", "rt", "rt", "rt…
#> $ lag                <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, …
#> $ model              <chr> "undefined", "undefined", "undefined", "undefined",…
#> $ interval_score     <dbl> 1.32269859, 1.09024833, 0.85173483, 0.62712459, 0.4…
#> $ dispersion         <dbl> 0.02613871, 0.02919685, 0.03194771, 0.03328885, 0.0…
#> $ underprediction    <dbl> 1.296559878, 1.061051473, 0.819787117, 0.593835733,…
#> $ overprediction     <dbl> 0.000000000, 0.000000000, 0.000000000, 0.000000000,…
#> $ coverage_deviation <dbl> -0.6714286, -0.6714286, -0.6714286, -0.6714286, -0.…
#> $ ae_median          <dbl> 1.507267490, 1.294189458, 1.071926118, 0.855356147,…
#> $ true_value         <dbl> 2.5, 2.5, 2.5, 2.5, 2.5, 2.5, 2.5, 2.5, 2.5, 2.5, 2…
#> $ crps               <dbl> 1.40846309, 1.18494251, 0.95493438, 0.73391430, 0.5…
#> $ bias               <dbl> -1.00000000, -0.99999975, -0.99998464, -0.99959928,…