Skip to contents

Container for Benchmarking Results

Source: R/BenchmarkResult.R
BenchmarkResult.Rd

This is the result container object returned by benchmark(). A BenchmarkResult consists of the data of multiple ResampleResults.

BenchmarkResults can be visualized via mlr3viz's autoplot() function.

For statistical analysis of benchmark results and more advanced plots, see mlr3benchmark.

Note

All stored objects are accessed by reference. Do not modify any extracted object without cloning it first.

S3 Methods

  • as.data.table(rr, ..., reassemble_learners = TRUE, convert_predictions = TRUE, predict_sets = "test")
    BenchmarkResult -> data.table::data.table()
    Returns a tabular view of the internal data.

  • c(...)
    (BenchmarkResult, ...) -> BenchmarkResult
    Combines multiple objects convertible to BenchmarkResult into a new BenchmarkResult.

See also

Other benchmark: benchmark(), benchmark_grid()

Active bindings

task_type

(character(1))
Task type of objects in the BenchmarkResult. All stored objects (Task, Learner, Prediction) in a single BenchmarkResult are required to have the same task type, e.g., "classif" or "regr". This is NA for empty BenchmarkResults.

tasks

(data.table::data.table())
Table of included Tasks with three columns:

  • "task_hash" (character(1)),

  • "task_id" (character(1)), and

  • "task" (Task).

learners

(data.table::data.table())
Table of included Learners with three columns:

  • "learner_hash" (character(1)),

  • "learner_id" (character(1)), and

  • "learner" (Learner).

Note that it is not feasible to access learned models via this field, as the training task would be ambiguous. For this reason the returned learner are reset before they are returned. Instead, select a row from the table returned by $score().

resamplings

(data.table::data.table())
Table of included Resamplings with three columns:

  • "resampling_hash" (character(1)),

  • "resampling_id" (character(1)), and

  • "resampling" (Resampling).

resample_results

(data.table::data.table())
Returns a table with three columns:

n_resample_results

(integer(1))
Returns the total number of stored ResampleResults.

uhashes

(character())
Set of (unique) hashes of all included ResampleResults.

Methods

Public methods

Method new()

Creates a new instance of this R6 class.

Usage

BenchmarkResult$new(data = NULL)

Arguments

data

(ResultData)
An object of type ResultData, either extracted from another ResampleResult, another BenchmarkResult, or manually constructed with as_result_data().

Method help()

Opens the help page for this object.

Usage

BenchmarkResult$help()

Method format()

Helper for print outputs.

Usage

BenchmarkResult$format(...)

Arguments

...

(ignored).

Method print()

Printer.

Usage

BenchmarkResult$print()

Method combine()

Fuses a second BenchmarkResult into itself, mutating the BenchmarkResult in-place. If the second BenchmarkResult bmr is NULL, simply returns self. Note that you can alternatively use the combine function c() which calls this method internally.

Usage

BenchmarkResult$combine(bmr)

Arguments

bmr

(BenchmarkResult)
A second BenchmarkResult object.

Returns

Returns the object itself, but modified by reference. You need to explicitly $clone() the object beforehand if you want to keep the object in its previous state.

Method score()

Returns a table with one row for each resampling iteration, including all involved objects: Task, Learner, Resampling, iteration number (integer(1)), and Prediction. If ids is set to TRUE, character column of extracted ids are added to the table for convenient filtering: "task_id", "learner_id", and "resampling_id".

Additionally calculates the provided performance measures and binds the performance scores as extra columns. These columns are named using the id of the respective Measure.

Usage

BenchmarkResult$score(
  measures = NULL,
  ids = TRUE,
  conditions = FALSE,
  predict_sets = "test"
)

Arguments

measures

(Measure | list of Measure)
Measure(s) to calculate.

ids

(logical(1))
Adds object ids ("task_id", "learner_id", "resampling_id") as extra character columns to the returned table.

conditions

(logical(1))
Adds condition messages ("warnings", "errors") as extra list columns of character vectors to the returned table

predict_sets

(character())
Prediction sets to operate on, used in aggregate() to extract the matching predict_sets from the ResampleResult. Multiple predict sets are calculated by the respective Learner during resample()/benchmark(). Must be a non-empty subset of {"train", "test", "holdout"}. If multiple sets are provided, these are first combined to a single prediction object. Default is "test".

Returns

data.table::data.table().

Method aggregate()

Returns a result table where resampling iterations are combined into ResampleResults. A column with the aggregated performance score is added for each Measure, named with the id of the respective measure.

The method for aggregation is controlled by the Measure, e.g. micro aggregation, macro aggregation or custom aggregation. Most measures default to macro aggregation.

Note that the aggregated performances just give a quick impression which approaches work well and which approaches are probably underperforming. However, the aggregates do not account for variance and cannot replace a statistical test. See mlr3viz to get a better impression via boxplots or mlr3benchmark for critical difference plots and significance tests.

For convenience, different flags can be set to extract more information from the returned ResampleResult.

Usage

BenchmarkResult$aggregate(
  measures = NULL,
  ids = TRUE,
  uhashes = FALSE,
  params = FALSE,
  conditions = FALSE
)

Arguments

measures

(Measure | list of Measure)
Measure(s) to calculate.

ids

(logical(1))
Adds object ids ("task_id", "learner_id", "resampling_id") as extra character columns for convenient subsetting.

uhashes

(logical(1))
Adds the uhash values of the ResampleResult as extra character column "uhash".

params

(logical(1))
Adds the hyperparameter values as extra list column "params". You can unnest them with mlr3misc::unnest().

conditions

(logical(1))
Adds the number of resampling iterations with at least one warning as extra integer column "warnings", and the number of resampling iterations with errors as extra integer column "errors".

Returns

data.table::data.table().

Method filter()

Subsets the benchmark result. If task_ids is not NULL, keeps all tasks with provided task ids and discards all others tasks. Same procedure for learner_ids and resampling_ids.

Usage

BenchmarkResult$filter(
  task_ids = NULL,
  task_hashes = NULL,
  learner_ids = NULL,
  learner_hashes = NULL,
  resampling_ids = NULL,
  resampling_hashes = NULL
)

Arguments

task_ids

(character())
Ids of Tasks to keep.

task_hashes

(character())
Hashes of Tasks to keep.

learner_ids

(character())
Ids of Learners to keep.

learner_hashes

(character())
Hashes of Learners to keep.

resampling_ids

(character())
Ids of Resamplings to keep.

resampling_hashes

(character())
Hashes of Resamplings to keep.

Returns

Returns the object itself, but modified by reference. You need to explicitly $clone() the object beforehand if you want to keeps the object in its previous state.

Method resample_result()

Retrieve the i-th ResampleResult, by position or by unique hash uhash. i and uhash are mutually exclusive.

Usage

BenchmarkResult$resample_result(i = NULL, uhash = NULL)

Arguments

i

(integer(1))
The iteration value to filter for.

uhash

(logical(1))
The ushash value to filter for.

Returns

ResampleResult.

Method discard()

Shrinks the BenchmarkResult by discarding parts of the internally stored data. Note that certain operations might stop work, e.g. extracting importance values from learners or calculating measures requiring the task's data.

Usage

BenchmarkResult$discard(backends = FALSE, models = FALSE)

Arguments

backends

(logical(1))
If TRUE, the DataBackend is removed from all stored Tasks.

models

(logical(1))
If TRUE, the stored model is removed from all Learners.

Returns

Returns the object itself, but modified by reference. You need to explicitly $clone() the object beforehand if you want to keeps the object in its previous state.

Method clone()

The objects of this class are cloneable with this method.

Usage

BenchmarkResult$clone(deep = FALSE)

Arguments

deep

Whether to make a deep clone.

Examples

set.seed(123)
learners = list(
  lrn("classif.featureless", predict_type = "prob"),
  lrn("classif.rpart", predict_type = "prob")
)

design = benchmark_grid(
  tasks = list(tsk("sonar"), tsk("penguins")),
  learners = learners,
  resamplings = rsmp("cv", folds = 3)
)
print(design)
#>        task             learner resampling
#>      <char>              <char>     <char>
#> 1:    sonar classif.featureless         cv
#> 2:    sonar       classif.rpart         cv
#> 3: penguins classif.featureless         cv
#> 4: penguins       classif.rpart         cv

bmr = benchmark(design)
print(bmr)
#> <BenchmarkResult> of 12 rows with 4 resampling runs
#>  nr  task_id          learner_id resampling_id iters warnings errors
#>   1    sonar classif.featureless            cv     3        0      0
#>   2    sonar       classif.rpart            cv     3        0      0
#>   3 penguins classif.featureless            cv     3        0      0
#>   4 penguins       classif.rpart            cv     3        0      0

bmr$tasks
#> Key: <task_hash>
#>           task_hash  task_id                   task
#>              <char>   <char>                 <list>
#> 1: 26bb097ecdef94ea penguins <TaskClassif:penguins>
#> 2: 76c4fc7a533d41b7    sonar    <TaskClassif:sonar>
bmr$learners
#> Key: <learner_hash>
#>        learner_hash          learner_id
#>              <char>              <char>
#> 1: 4a8ddb31f4547017       classif.rpart
#> 2: 54ecc16720ea1edf classif.featureless
#>                                            learner
#>                                             <list>
#> 1:             <LearnerClassifRpart:classif.rpart>
#> 2: <LearnerClassifFeatureless:classif.featureless>

# first 5 resampling iterations
head(as.data.table(bmr, measures = c("classif.acc", "classif.auc")), 5)
#>                                   uhash                task
#>                                  <char>              <list>
#> 1: 6bd5fc78-43b7-486c-9b8c-f92228b42188 <TaskClassif:sonar>
#> 2: 6bd5fc78-43b7-486c-9b8c-f92228b42188 <TaskClassif:sonar>
#> 3: 6bd5fc78-43b7-486c-9b8c-f92228b42188 <TaskClassif:sonar>
#> 4: b01bf8fc-c2ef-4bf2-b707-2fa8eccb9102 <TaskClassif:sonar>
#> 5: b01bf8fc-c2ef-4bf2-b707-2fa8eccb9102 <TaskClassif:sonar>
#>                                            learner     resampling iteration
#>                                             <list>         <list>     <int>
#> 1: <LearnerClassifFeatureless:classif.featureless> <ResamplingCV>         1
#> 2: <LearnerClassifFeatureless:classif.featureless> <ResamplingCV>         2
#> 3: <LearnerClassifFeatureless:classif.featureless> <ResamplingCV>         3
#> 4:             <LearnerClassifRpart:classif.rpart> <ResamplingCV>         1
#> 5:             <LearnerClassifRpart:classif.rpart> <ResamplingCV>         2
#>             prediction
#>                 <list>
#> 1: <PredictionClassif>
#> 2: <PredictionClassif>
#> 3: <PredictionClassif>
#> 4: <PredictionClassif>
#> 5: <PredictionClassif>

# aggregate results
bmr$aggregate()
#>       nr  task_id          learner_id resampling_id iters classif.ce
#>    <int>   <char>              <char>        <char> <int>      <num>
#> 1:     1    sonar classif.featureless            cv     3 0.46604555
#> 2:     2    sonar       classif.rpart            cv     3 0.27391304
#> 3:     3 penguins classif.featureless            cv     3 0.55814900
#> 4:     4 penguins       classif.rpart            cv     3 0.05812357
#> Hidden columns: resample_result

# aggregate results with hyperparameters as separate columns
mlr3misc::unnest(bmr$aggregate(params = TRUE), "params")
#>       nr  task_id          learner_id resampling_id iters classif.ce method
#>    <int>   <char>              <char>        <char> <int>      <num> <char>
#> 1:     1    sonar classif.featureless            cv     3 0.46604555   mode
#> 2:     2    sonar       classif.rpart            cv     3 0.27391304   <NA>
#> 3:     3 penguins classif.featureless            cv     3 0.55814900   mode
#> 4:     4 penguins       classif.rpart            cv     3 0.05812357   <NA>
#>     xval
#>    <int>
#> 1:    NA
#> 2:     0
#> 3:    NA
#> 4:     0
#> Hidden columns: resample_result

# extract resample result for classif.rpart
rr = bmr$aggregate()[learner_id == "classif.rpart", resample_result][[1]]
print(rr)
#> <ResampleResult> with 3 resampling iterations
#>  task_id    learner_id resampling_id iteration warnings errors
#>    sonar classif.rpart            cv         1        0      0
#>    sonar classif.rpart            cv         2        0      0
#>    sonar classif.rpart            cv         3        0      0

# access the confusion matrix of the first resampling iteration
rr$predictions()[[1]]$confusion
#>         truth
#> response  M  R
#>        M 30 18
#>        R  3 19

# reduce to subset with task id "sonar"
bmr$filter(task_ids = "sonar")
print(bmr)
#> <BenchmarkResult> of 6 rows with 2 resampling runs
#>  nr task_id          learner_id resampling_id iters warnings errors
#>   1   sonar classif.featureless            cv     3        0      0
#>   2   sonar       classif.rpart            cv     3        0      0