Runs a resampling (possibly in parallel).

resample(task, learner, resampling, store_models = FALSE)

Arguments

task

:: Task.

learner

:: Learner.

resampling

:: Resampling.

store_models

:: logical(1)
Keep the fitted model after the test set has been predicted? Set to TRUE if you want to further analyse the models or want to extract information like variable importance.

Value

ResampleResult.

Note

The fitted models are discarded after the predictions have been scored in order to reduce memory consumption. If you need access to the models for later analysis, set store_models to TRUE.

Parallelization

This function can be parallelized with the future package. One job is one resampling iteration, and all jobs are send to an apply function from future.apply in a single batch. To select a parallel backend, use future::plan().

Logging

The mlr3 uses the lgr package for logging. lgr supports multiple log levels which can be queried with getOption("lgr.log_levels").

To suppress output and reduce verbosity, you can lower the log from the default level "info" to "warn":

lgr::get_logger("mlr3")$set_threshold("warn")

To get additional log output for debugging, increase the log level to "debug" or "trace":

lgr::get_logger("mlr3")$set_threshold("debug")

To log to a file or a data base, see the documentation of lgr::lgr-package.

Examples

task = tsk("iris") learner = lrn("classif.rpart") resampling = rsmp("cv") # explicitly instantiate the resampling for this task for reproduciblity set.seed(123) resampling$instantiate(task) rr = resample(task, learner, resampling) print(rr)
#> <ResampleResult> of 10 iterations #> * Task: iris #> * Learner: classif.rpart #> * Warnings: 0 in 0 iterations #> * Errors: 0 in 0 iterations
# retrieve performance rr$score(msr("classif.ce"))
#> task task_id learner learner_id resampling #> 1: <TaskClassif> iris <LearnerClassifRpart> classif.rpart <ResamplingCV> #> 2: <TaskClassif> iris <LearnerClassifRpart> classif.rpart <ResamplingCV> #> 3: <TaskClassif> iris <LearnerClassifRpart> classif.rpart <ResamplingCV> #> 4: <TaskClassif> iris <LearnerClassifRpart> classif.rpart <ResamplingCV> #> 5: <TaskClassif> iris <LearnerClassifRpart> classif.rpart <ResamplingCV> #> 6: <TaskClassif> iris <LearnerClassifRpart> classif.rpart <ResamplingCV> #> 7: <TaskClassif> iris <LearnerClassifRpart> classif.rpart <ResamplingCV> #> 8: <TaskClassif> iris <LearnerClassifRpart> classif.rpart <ResamplingCV> #> 9: <TaskClassif> iris <LearnerClassifRpart> classif.rpart <ResamplingCV> #> 10: <TaskClassif> iris <LearnerClassifRpart> classif.rpart <ResamplingCV> #> resampling_id iteration prediction classif.ce #> 1: cv 1 <list> 0.00000000 #> 2: cv 2 <list> 0.20000000 #> 3: cv 3 <list> 0.20000000 #> 4: cv 4 <list> 0.06666667 #> 5: cv 5 <list> 0.06666667 #> 6: cv 6 <list> 0.00000000 #> 7: cv 7 <list> 0.00000000 #> 8: cv 8 <list> 0.00000000 #> 9: cv 9 <list> 0.06666667 #> 10: cv 10 <list> 0.00000000
rr$aggregate(msr("classif.ce"))
#> classif.ce #> 0.06
# merged prediction objects of all resampling iterations pred = rr$prediction() pred$confusion
#> truth #> response setosa versicolor virginica #> setosa 50 0 0 #> versicolor 0 46 5 #> virginica 0 4 45
# Repeat resampling with featureless learner rr_featureless = resample(task, lrn("classif.featureless"), resampling) # Convert results to BenchmarkResult, then combine them bmr1 = as_benchmark_result(rr) bmr2 = as_benchmark_result(rr_featureless) print(bmr1$combine(bmr2))
#> <BenchmarkResult> of 20 rows with 2 resampling runs #> nr task_id learner_id resampling_id iters warnings errors #> 1 iris classif.rpart cv 10 0 0 #> 2 iris classif.featureless cv 10 0 0