Skip to contents

Takes a lists of Task, a list of Learner and a list of Resampling to generate a design in an expand.grid() fashion (a.k.a. cross join or Cartesian product).

Resampling strategies are not allowed to be instantiated when passing the argument, and instead will be instantiated per task internally. The only exception to this rule applies if all tasks have exactly the same number of rows, and the resamplings are all instantiated for such tasks.

Usage

benchmark_grid(tasks, learners, resamplings)

Arguments

tasks

(list of Task).

learners

(list of Learner).

resamplings

(list of Resampling).

Value

(data.table::data.table()) with the cross product of the input vectors.

See also

Other benchmark: BenchmarkResult, benchmark()

Examples

tasks = list(tsk("penguins"), tsk("sonar"))
learners = list(lrn("classif.featureless"), lrn("classif.rpart"))
resamplings = list(rsmp("cv"), rsmp("subsampling"))

grid = benchmark_grid(tasks, learners, resamplings)
print(grid)
#>        task             learner  resampling
#> 1: penguins classif.featureless          cv
#> 2: penguins classif.featureless subsampling
#> 3: penguins       classif.rpart          cv
#> 4: penguins       classif.rpart subsampling
#> 5:    sonar classif.featureless          cv
#> 6:    sonar classif.featureless subsampling
#> 7:    sonar       classif.rpart          cv
#> 8:    sonar       classif.rpart subsampling
if (FALSE) {
benchmark(grid)
}

# manual construction of the grid with data.table::CJ()
grid = data.table::CJ(task = tasks, learner = learners,
  resampling = resamplings, sorted = FALSE)

# manual instantiation (not suited for a fair comparison of learners!)
Map(function(task, resampling) {
  resampling$instantiate(task)
}, task = grid$task, resampling = grid$resampling)
#> [[1]]
#> <ResamplingCV>: Cross-Validation
#> * Iterations: 10
#> * Instantiated: TRUE
#> * Parameters: folds=10
#> 
#> [[2]]
#> <ResamplingSubsampling>: Subsampling
#> * Iterations: 30
#> * Instantiated: TRUE
#> * Parameters: repeats=30, ratio=0.6667
#> 
#> [[3]]
#> <ResamplingCV>: Cross-Validation
#> * Iterations: 10
#> * Instantiated: TRUE
#> * Parameters: folds=10
#> 
#> [[4]]
#> <ResamplingSubsampling>: Subsampling
#> * Iterations: 30
#> * Instantiated: TRUE
#> * Parameters: repeats=30, ratio=0.6667
#> 
#> [[5]]
#> <ResamplingCV>: Cross-Validation
#> * Iterations: 10
#> * Instantiated: TRUE
#> * Parameters: folds=10
#> 
#> [[6]]
#> <ResamplingSubsampling>: Subsampling
#> * Iterations: 30
#> * Instantiated: TRUE
#> * Parameters: repeats=30, ratio=0.6667
#> 
#> [[7]]
#> <ResamplingCV>: Cross-Validation
#> * Iterations: 10
#> * Instantiated: TRUE
#> * Parameters: folds=10
#> 
#> [[8]]
#> <ResamplingSubsampling>: Subsampling
#> * Iterations: 30
#> * Instantiated: TRUE
#> * Parameters: repeats=30, ratio=0.6667
#> 
if (FALSE) {
benchmark(grid)
}