Takes a lists of Task, a list of Learner and a list of Resampling to
generate a design in an expand.grid() fashion (a.k.a. cross join or Cartesian product).
Resampling strategies are not allowed to be instantiated when passing the argument, and instead will be instantiated per task internally. The only exception to this rule applies if all tasks have exactly the same number of rows, and the resamplings are all instantiated for such tasks.
Arguments
- tasks
(list of Task).
- learners
(list of Learner).
- resamplings
(list of Resampling).
Value
(data.table::data.table()) with the cross product of the input vectors.
See also
Chapter in the mlr3book: https://mlr3book.mlr-org.com/performance.html
Package mlr3viz for some generic visualizations.
mlr3benchmark for post-hoc analysis of benchmark results.
Other benchmark:
BenchmarkResult,
benchmark()
Examples
tasks = list(tsk("penguins"), tsk("sonar"))
learners = list(lrn("classif.featureless"), lrn("classif.rpart"))
resamplings = list(rsmp("cv"), rsmp("subsampling"))
grid = benchmark_grid(tasks, learners, resamplings)
print(grid)
#> task learner resampling
#> 1: penguins classif.featureless cv
#> 2: penguins classif.featureless subsampling
#> 3: penguins classif.rpart cv
#> 4: penguins classif.rpart subsampling
#> 5: sonar classif.featureless cv
#> 6: sonar classif.featureless subsampling
#> 7: sonar classif.rpart cv
#> 8: sonar classif.rpart subsampling
if (FALSE) {
benchmark(grid)
}
# manual construction of the grid with data.table::CJ()
grid = data.table::CJ(task = tasks, learner = learners,
resampling = resamplings, sorted = FALSE)
# manual instantiation (not suited for a fair comparison of learners!)
Map(function(task, resampling) {
resampling$instantiate(task)
}, task = grid$task, resampling = grid$resampling)
#> [[1]]
#> <ResamplingCV>: Cross-Validation
#> * Iterations: 10
#> * Instantiated: TRUE
#> * Parameters: folds=10
#>
#> [[2]]
#> <ResamplingSubsampling>: Subsampling
#> * Iterations: 30
#> * Instantiated: TRUE
#> * Parameters: repeats=30, ratio=0.6667
#>
#> [[3]]
#> <ResamplingCV>: Cross-Validation
#> * Iterations: 10
#> * Instantiated: TRUE
#> * Parameters: folds=10
#>
#> [[4]]
#> <ResamplingSubsampling>: Subsampling
#> * Iterations: 30
#> * Instantiated: TRUE
#> * Parameters: repeats=30, ratio=0.6667
#>
#> [[5]]
#> <ResamplingCV>: Cross-Validation
#> * Iterations: 10
#> * Instantiated: TRUE
#> * Parameters: folds=10
#>
#> [[6]]
#> <ResamplingSubsampling>: Subsampling
#> * Iterations: 30
#> * Instantiated: TRUE
#> * Parameters: repeats=30, ratio=0.6667
#>
#> [[7]]
#> <ResamplingCV>: Cross-Validation
#> * Iterations: 10
#> * Instantiated: TRUE
#> * Parameters: folds=10
#>
#> [[8]]
#> <ResamplingSubsampling>: Subsampling
#> * Iterations: 30
#> * Instantiated: TRUE
#> * Parameters: repeats=30, ratio=0.6667
#>
if (FALSE) {
benchmark(grid)
}