Runs a resampling (possibly in parallel).

resample(task, learner, resampling, ctrl = list())

Arguments

task

:: Task.

learner

:: Learner.

resampling

:: Resampling.

ctrl

:: named list()
Object to control learner execution. See mlr_control() for details.

Value

ResampleResult.

Note

The fitted models are discarded after the predictions have been scored in order to reduce memory consumption. If you need access to the models for later analysis, set store_models to TRUE via mlr_control().

Syntactic Sugar

The mlr3 package provides some shortcuts to ease the creation of its objects.

First, instead of the objects themselves, it is possible to pass a character() vector which is used to lookup the provided keys in a mlr3misc::Dictonary:

Additionally, each task has an associated default measure (stored in mlr_reflections) which is used as a fallback if no other measure is provided. Classification tasks default to the classification error in "classif.ce", regression tasks to the mean squared error in "regr.mse".

Parallelization

This function can be parallelized with the future package. One job is one resampling iteration, and all jobs are send to an apply function from future.apply in a single batch. To select a parallel backend, use future::plan().

Examples

task = mlr_tasks$get("iris") learner = mlr_learners$get("classif.rpart") resampling = mlr_resamplings$get("cv") # explicitly instantiate the resampling for this task for reproduciblity set.seed(123) resampling$instantiate(task) rr = resample(task, learner, resampling) print(rr)
#> <ResampleResult> of 10 iterations #> Task: iris #> Learner: classif.rpart
# retrieve performance rr$performance("classif.ce")
#> task task_id learner learner_id resampling #> 1: <TaskClassif> iris <LearnerClassifRpart> classif.rpart <ResamplingCV> #> 2: <TaskClassif> iris <LearnerClassifRpart> classif.rpart <ResamplingCV> #> 3: <TaskClassif> iris <LearnerClassifRpart> classif.rpart <ResamplingCV> #> 4: <TaskClassif> iris <LearnerClassifRpart> classif.rpart <ResamplingCV> #> 5: <TaskClassif> iris <LearnerClassifRpart> classif.rpart <ResamplingCV> #> 6: <TaskClassif> iris <LearnerClassifRpart> classif.rpart <ResamplingCV> #> 7: <TaskClassif> iris <LearnerClassifRpart> classif.rpart <ResamplingCV> #> 8: <TaskClassif> iris <LearnerClassifRpart> classif.rpart <ResamplingCV> #> 9: <TaskClassif> iris <LearnerClassifRpart> classif.rpart <ResamplingCV> #> 10: <TaskClassif> iris <LearnerClassifRpart> classif.rpart <ResamplingCV> #> resampling_id iteration prediction classif.ce #> 1: cv 1 <PredictionClassif> 0.00000000 #> 2: cv 2 <PredictionClassif> 0.20000000 #> 3: cv 3 <PredictionClassif> 0.20000000 #> 4: cv 4 <PredictionClassif> 0.06666667 #> 5: cv 5 <PredictionClassif> 0.06666667 #> 6: cv 6 <PredictionClassif> 0.00000000 #> 7: cv 7 <PredictionClassif> 0.00000000 #> 8: cv 8 <PredictionClassif> 0.00000000 #> 9: cv 9 <PredictionClassif> 0.06666667 #> 10: cv 10 <PredictionClassif> 0.00000000
rr$aggregate("classif.ce")
#> classif.ce #> 0.06
# merged prediction objects of all resampling iterations pred = rr$prediction pred$confusion
#> truth #> response setosa versicolor virginica #> setosa 50 0 0 #> versicolor 0 46 5 #> virginica 0 4 45
# Repeat resampling with featureless learner rr.featureless = resample(task, "classif.featureless", resampling) # Combine the ResampleResults into a BenchmarkResult bmr = rr$combine(rr.featureless) print(bmr)
#> <BenchmarkResult> of 20 rows with 2 resampling runs