# mlr3

Package website: [release](https://mlr3.mlr-org.com/) \|
[dev](https://mlr3.mlr-org.com/dev/)

Efficient, object-oriented programming on the building blocks of machine
learning. Successor of [mlr](https://github.com/mlr-org/mlr).

## Resources (for users and developers)

- We have written a [book](https://mlr3book.mlr-org.com/). This should
  be the central entry point to the package.
- The [mlr-org website](https://mlr-org.com/) includes for example a
  [gallery](https://mlr-org.com/gallery.html) with case studies.
- [Reference manual](https://mlr3.mlr-org.com/reference/)
- [FAQ](https://mlr-org.com/faq.html)
- Ask questions on Stackoverflow (tag mlr3)
- **Extension Learners**
  - Recommended core regression, classification, and survival learners
    are in [mlr3learners](https://github.com/mlr-org/mlr3learners)
  - All others are in
    [mlr3extralearners](https://github.com/mlr-org/mlr3extralearners)
  - Use the [learner search](https://mlr-org.com/learners.html) to get a
    simple overview
- **Cheatsheets**
  - [Overview of cheatsheets](https://cheatsheets.mlr-org.com)
  - [mlr3](https://cheatsheets.mlr-org.com/mlr3.pdf)
  - [mlr3tuning](https://cheatsheets.mlr-org.com/mlr3tuning.pdf)
  - [mlr3pipelines](https://cheatsheets.mlr-org.com/mlr3pipelines.pdf)
- **Videos**:
  - [useR2019 talk on mlr3](https://www.youtube.com/watch?v=wsP2hiFnDQs)
  - [useR2019 talk on mlr3pipelines and
    mlr3tuning](https://www.youtube.com/watch?v=gEW5RxkbQuQ)
  - [useR2020 tutorial on mlr3, mlr3tuning and
    mlr3pipelines](https://www.youtube.com/watch?v=T43hO2o_nZw)
- **Courses/Lectures**
  - The course [Introduction to Machine learning
    (I2ML)](https://slds-lmu.github.io/i2ml/) is a free and open flipped
    classroom course on the basics of machine learning. `mlr3` is used
    in the
    [demos](https://github.com/slds-lmu/lecture_i2ml/tree/master/code-demos-pdf)
    and
    [exercises](https://github.com/slds-lmu/lecture_i2ml/tree/master/exercises).
- **Templates/Tutorials**
  - [mlr3-targets](https://github.com/mlr-org/mlr3-targets): Tutorial
    showcasing how to use {mlr3} with
    [targets](https://docs.ropensci.org/targets/) for reproducible ML
    workflow automation.
  - Using `mlr3viz` and `animint2` for [interactive
    visualizations](https://animint-manual-en.netlify.app/ch20/).
- [List of extension packages](https://mlr-org.com/ecosystem.html)
- [mlr-outreach](https://github.com/mlr-org/mlr-outreach) contains
  public talks and slides resources.
- [Wiki](https://github.com/mlr-org/mlr3/wiki): Contains mainly
  information for developers.

## Installation

Install the last release from CRAN:

``` r

install.packages("mlr3")
```

Install the development version from GitHub:

``` r

# install.packages("pak")
pak::pak("mlr-org/mlr3")
```

If you want to get started with `mlr3`, we recommend installing the
[mlr3verse](https://mlr3verse.mlr-org.com/) meta-package which installs
`mlr3` and some of the most important extension packages:

``` r

install.packages("mlr3verse")
```

## Example

### Constructing Learners and Tasks

``` r

library(mlr3)

# create learning task
task_penguins = as_task_classif(species ~ ., data = palmerpenguins::penguins)
task_penguins
```

``` R
## 
## ── <TaskClassif> (344x8) ───────────────────────────────────────────────────────
## • Target: species
## • Target classes: Adelie (44%), Gentoo (36%), Chinstrap (20%)
## • Properties: multiclass
## • Features (7):
##   • int (3): body_mass_g, flipper_length_mm, year
##   • dbl (2): bill_depth_mm, bill_length_mm
##   • fct (2): island, sex
```

``` r

# load learner and set hyperparameter
learner = lrn("classif.rpart", cp = .01)
```

### Basic train + predict

``` r

# train/test split
split = partition(task_penguins, ratio = 0.67)

# train the model
learner$train(task_penguins, split$train_set)

# predict data
prediction = learner$predict(task_penguins, split$test_set)

# calculate performance
prediction$confusion
```

``` R
##            truth
## response    Adelie Chinstrap Gentoo
##   Adelie       146         5      0
##   Chinstrap      6        63      1
##   Gentoo         0         0    123
```

``` r

measure = msr("classif.acc")
prediction$score(measure)
```

``` R
## classif.acc 
##   0.9651163
```

### Resample

``` r

# 3-fold cross validation
resampling = rsmp("cv", folds = 3L)

# run experiments
rr = resample(task_penguins, learner, resampling)

# access results
rr$score(measure)[, .(task_id, learner_id, iteration, classif.acc)]
```

``` R
##                     task_id    learner_id iteration classif.acc
## 1: palmerpenguins::penguins classif.rpart         1   0.8956522
## 2: palmerpenguins::penguins classif.rpart         2   0.9478261
## 3: palmerpenguins::penguins classif.rpart         3   0.9649123
```

``` r

rr$aggregate(measure)
```

``` R
## classif.acc 
##   0.9361302
```

## Extension Packages

[![](reference/figures/mlr3verse.svg)](https://raw.githubusercontent.com/mlr-org/mlr3/main/man/figures/mlr3verse.svg?sanitize=true)

Consult the
[wiki](https://github.com/mlr-org/mlr3/wiki/Extension-Packages) for
short descriptions and links to the respective repositories.

For beginners, we strongly recommend to install and load the
[mlr3verse](https://mlr3verse.mlr-org.com/) package for a better user
experience.

## Why a rewrite?

[mlr](https://github.com/mlr-org/mlr) was first released to
[CRAN](https://cran.r-project.org/package=mlr) in 2013. Its core design
and architecture date back even further. The addition of many features
has led to a [feature
creep](https://en.wikipedia.org/wiki/Feature_creep) which makes
[mlr](https://github.com/mlr-org/mlr) hard to maintain and hard to
extend. We also think that while mlr was nicely extensible in some parts
(learners, measures, etc.), other parts were less easy to extend from
the outside. Also, many helpful R libraries did not exist at the time
[mlr](https://github.com/mlr-org/mlr) was created, and their inclusion
would result in non-trivial API changes.

## Design principles

- Only the basic building blocks for machine learning are implemented in
  this package.
- Focus on computation here. No visualization or other stuff. That can
  go in extra packages.
- Overcome the limitations of R’s [S3
  classes](https://adv-r.hadley.nz/s3.html) with the help of
  [R6](https://cran.r-project.org/package=R6).
- Embrace [R6](https://cran.r-project.org/package=R6) for a clean
  OO-design, object state-changes and reference semantics. This might be
  less “traditional R”, but seems to fit `mlr` nicely.
- Embrace [`data.table`](https://cran.r-project.org/package=data.table)
  for fast and convenient data frame computations.
- Combine `data.table` and `R6`, for this we will make heavy use of list
  columns in data.tables.
- Defensive programming and type safety. All user input is checked with
  [`checkmate`](https://cran.r-project.org/package=checkmate). Return
  types are documented, and mechanisms popular in base R which
  “simplify” the result unpredictably (e.g.,
  [`sapply()`](https://rdrr.io/r/base/lapply.html) or `drop` argument in
  `[.data.frame`) are avoided.
- Be light on dependencies. `mlr3` requires the following packages at
  runtime:
  - [`parallelly`](https://cran.r-project.org/package=parallelly):
    Helper functions for parallelization. No extra recursive
    dependencies.
  - [`future.apply`](https://cran.r-project.org/package=future.apply):
    Resampling and benchmarking is parallelized with the
    [`future`](https://cran.r-project.org/package=future) abstraction
    interfacing many parallel backends.
  - [`backports`](https://cran.r-project.org/package=backports): Ensures
    backward compatibility with older R releases. Developed by members
    of the `mlr` team. No recursive dependencies.
  - [`checkmate`](https://cran.r-project.org/package=checkmate): Fast
    argument checks. Developed by members of the `mlr` team. No extra
    recursive dependencies.
  - [`mlr3misc`](https://cran.r-project.org/package=mlr3misc):
    Miscellaneous functions used in multiple mlr3 [extension
    packages](https://mlr-org.com/ecosystem.html). Developed by the
    `mlr` team.
  - [`paradox`](https://cran.r-project.org/package=paradox):
    Descriptions for parameters and parameter sets. Developed by the
    `mlr` team. No extra recursive dependencies.
  - [`R6`](https://cran.r-project.org/package=R6): Reference class
    objects. No recursive dependencies.
  - [`data.table`](https://cran.r-project.org/package=data.table):
    Extension of R’s `data.frame`. No recursive dependencies.
  - [`digest`](https://cran.r-project.org/package=digest) (via
    `mlr3misc`): Hash digests. No recursive dependencies.
  - [`uuid`](https://cran.r-project.org/package=uuid): Create unique
    string identifiers. No recursive dependencies.
  - [`lgr`](https://cran.r-project.org/package=lgr): Logging facility.
    No extra recursive dependencies.
  - [`mlr3measures`](https://cran.r-project.org/package=mlr3measures):
    Performance measures. No extra recursive dependencies.
  - [`mlbench`](https://cran.r-project.org/package=mlbench): A
    collection of machine learning data sets. No dependencies.
  - [`palmerpenguins`](https://cran.r-project.org/package=palmerpenguins):
    A classification data set about penguins, used on examples and
    provided as a toy task. No dependencies.
- [Reflections](https://en.wikipedia.org/wiki/Reflection_%28computer_programming%29):
  Objects are queryable for properties and capabilities, allowing you to
  program on them.
- Additional functionality that comes with extra dependencies:
  - To capture output, warnings and exceptions,
    [`evaluate`](https://cran.r-project.org/package=evaluate) and
    [`callr`](https://cran.r-project.org/package=callr) can be used.

## Contributing to mlr3

This R package is licensed under the
[LGPL-3](https://www.gnu.org/licenses/lgpl-3.0.html.en). If you
encounter problems using this software (lack of documentation,
misleading or wrong documentation, unexpected behavior, bugs, …) or just
want to suggest features, please open an issue in the [issue
tracker](https://github.com/mlr-org/mlr3/issues). Pull requests are
welcome and will be included at the discretion of the maintainers.

Please consult the [wiki](https://github.com/mlr-org/mlr3/wiki/) for a
[style guide](https://github.com/mlr-org/mlr3/wiki/Style-Guide), a
[roxygen guide](https://github.com/mlr-org/mlr3/wiki/Roxygen-Guide) and
a [pull request
guide](https://github.com/mlr-org/mlr3/wiki/PR-Guidelines).

## Citing mlr3

If you use mlr3, please cite our [JOSS
article](https://doi.org/10.21105/joss.01903):

``` R
@Article{mlr3,
  title = {{mlr3}: A modern object-oriented machine learning framework in {R}},
  author = {Michel Lang and Martin Binder and Jakob Richter and Patrick Schratz and Florian Pfisterer and Stefan Coors and Quay Au and Giuseppe Casalicchio and Lars Kotthoff and Bernd Bischl},
  journal = {Journal of Open Source Software},
  year = {2019},
  month = {dec},
  doi = {10.21105/joss.01903},
  url = {https://joss.theoj.org/papers/10.21105/joss.01903},
}
```

# Package index

## Building Blocks

- [`Task`](https://mlr3.mlr-org.com/reference/Task.md) : Task Class
- [`Learner`](https://mlr3.mlr-org.com/reference/Learner.md) : Learner
  Class
- [`Measure`](https://mlr3.mlr-org.com/reference/Measure.md) : Measure
  Class
- [`Resampling`](https://mlr3.mlr-org.com/reference/Resampling.md) :
  Resampling Class
- [`Prediction`](https://mlr3.mlr-org.com/reference/Prediction.md) :
  Abstract Prediction Object

## Data Backends

- [`DataBackend`](https://mlr3.mlr-org.com/reference/DataBackend.md) :
  DataBackend
- [`DataBackendDataTable`](https://mlr3.mlr-org.com/reference/DataBackendDataTable.md)
  : DataBackend for data.table
- [`as_data_backend()`](https://mlr3.mlr-org.com/reference/as_data_backend.md)
  : Create a Data Backend

## Classification

- [`TaskClassif`](https://mlr3.mlr-org.com/reference/TaskClassif.md) :
  Classification Task
- [`as_task_classif()`](https://mlr3.mlr-org.com/reference/as_task_classif.md)
  : Convert to a Classification Task
- [`LearnerClassif`](https://mlr3.mlr-org.com/reference/LearnerClassif.md)
  : Classification Learner
- [`MeasureClassif`](https://mlr3.mlr-org.com/reference/MeasureClassif.md)
  : Classification Measure
- [`PredictionClassif`](https://mlr3.mlr-org.com/reference/PredictionClassif.md)
  : Prediction Object for Classification

## Regression

- [`TaskRegr`](https://mlr3.mlr-org.com/reference/TaskRegr.md) :
  Regression Task
- [`as_task_regr()`](https://mlr3.mlr-org.com/reference/as_task_regr.md)
  : Convert to a Regression Task
- [`LearnerRegr`](https://mlr3.mlr-org.com/reference/LearnerRegr.md) :
  Regression Learner
- [`MeasureRegr`](https://mlr3.mlr-org.com/reference/MeasureRegr.md) :
  Regression Measure
- [`PredictionRegr`](https://mlr3.mlr-org.com/reference/PredictionRegr.md)
  : Prediction Object for Regression

## Tasks

- [`california_housing`](https://mlr3.mlr-org.com/reference/california_housing.md)
  [`mlr_tasks_california_housing`](https://mlr3.mlr-org.com/reference/california_housing.md)
  : Median House Value in California
- [`mlr_tasks`](https://mlr3.mlr-org.com/reference/mlr_tasks.md) :
  Dictionary of Tasks
- [`mlr_tasks_breast_cancer`](https://mlr3.mlr-org.com/reference/mlr_tasks_breast_cancer.md)
  : Wisconsin Breast Cancer Classification Task
- [`mlr_tasks_german_credit`](https://mlr3.mlr-org.com/reference/mlr_tasks_german_credit.md)
  : German Credit Classification Task
- [`mlr_tasks_iris`](https://mlr3.mlr-org.com/reference/mlr_tasks_iris.md)
  : Iris Classification Task
- [`mlr_tasks_mtcars`](https://mlr3.mlr-org.com/reference/mlr_tasks_mtcars.md)
  : Motor Trend Regression Task
- [`mlr_tasks_penguins`](https://mlr3.mlr-org.com/reference/mlr_tasks_penguins.md)
  : Palmer Penguins Data Set
- [`mlr_tasks_pima`](https://mlr3.mlr-org.com/reference/mlr_tasks_pima.md)
  : Pima Indian Diabetes Classification Task
- [`mlr_tasks_sonar`](https://mlr3.mlr-org.com/reference/mlr_tasks_sonar.md)
  : Sonar Classification Task
- [`mlr_tasks_spam`](https://mlr3.mlr-org.com/reference/mlr_tasks_spam.md)
  : Spam Classification Task
- [`mlr_tasks_wine`](https://mlr3.mlr-org.com/reference/mlr_tasks_wine.md)
  : Wine Classification Task
- [`mlr_tasks_zoo`](https://mlr3.mlr-org.com/reference/mlr_tasks_zoo.md)
  : Zoo Classification Task
- [`as_task()`](https://mlr3.mlr-org.com/reference/as_task.md)
  [`as_tasks()`](https://mlr3.mlr-org.com/reference/as_task.md) :
  Convert to a Task
- [`convert_task()`](https://mlr3.mlr-org.com/reference/convert_task.md)
  : Convert a Task from One Type to Another

## Task Generators

- [`TaskGenerator`](https://mlr3.mlr-org.com/reference/TaskGenerator.md)
  : TaskGenerator Class
- [`mlr_task_generators`](https://mlr3.mlr-org.com/reference/mlr_task_generators.md)
  : Dictionary of Task Generators
- [`mlr_task_generators_2dnormals`](https://mlr3.mlr-org.com/reference/mlr_task_generators_2dnormals.md)
  [`TaskGenerator2DNormals`](https://mlr3.mlr-org.com/reference/mlr_task_generators_2dnormals.md)
  : 2D Normals Classification Task Generator
- [`mlr_task_generators_cassini`](https://mlr3.mlr-org.com/reference/mlr_task_generators_cassini.md)
  [`TaskGeneratorCassini`](https://mlr3.mlr-org.com/reference/mlr_task_generators_cassini.md)
  : Cassini Classification Task Generator
- [`mlr_task_generators_circle`](https://mlr3.mlr-org.com/reference/mlr_task_generators_circle.md)
  [`TaskGeneratorCircle`](https://mlr3.mlr-org.com/reference/mlr_task_generators_circle.md)
  : Circle Classification Task Generator
- [`mlr_task_generators_friedman1`](https://mlr3.mlr-org.com/reference/mlr_task_generators_friedman1.md)
  [`TaskGeneratorFriedman1`](https://mlr3.mlr-org.com/reference/mlr_task_generators_friedman1.md)
  : Friedman1 Regression Task Generator
- [`mlr_task_generators_moons`](https://mlr3.mlr-org.com/reference/mlr_task_generators_moons.md)
  [`TaskGeneratorMoons`](https://mlr3.mlr-org.com/reference/mlr_task_generators_moons.md)
  : Moons Classification Task Generator
- [`mlr_task_generators_peak`](https://mlr3.mlr-org.com/reference/mlr_task_generators_peak.md)
  [`TaskGeneratorPeak`](https://mlr3.mlr-org.com/reference/mlr_task_generators_peak.md)
  : Peak Regression Task Generator
- [`mlr_task_generators_simplex`](https://mlr3.mlr-org.com/reference/mlr_task_generators_simplex.md)
  [`TaskGeneratorSimplex`](https://mlr3.mlr-org.com/reference/mlr_task_generators_simplex.md)
  : Simplex Classification Task Generator
- [`mlr_task_generators_smiley`](https://mlr3.mlr-org.com/reference/mlr_task_generators_smiley.md)
  [`TaskGeneratorSmiley`](https://mlr3.mlr-org.com/reference/mlr_task_generators_smiley.md)
  : Smiley Classification Task Generator
- [`mlr_task_generators_spirals`](https://mlr3.mlr-org.com/reference/mlr_task_generators_spirals.md)
  [`TaskGeneratorSpirals`](https://mlr3.mlr-org.com/reference/mlr_task_generators_spirals.md)
  : Spiral Classification Task Generator
- [`mlr_task_generators_xor`](https://mlr3.mlr-org.com/reference/mlr_task_generators_xor.md)
  [`TaskGeneratorXor`](https://mlr3.mlr-org.com/reference/mlr_task_generators_xor.md)
  : XOR Classification Task Generator

## Learners

- [`mlr_learners`](https://mlr3.mlr-org.com/reference/mlr_learners.md) :
  Dictionary of Learners
- [`mlr_learners_classif.debug`](https://mlr3.mlr-org.com/reference/mlr_learners_classif.debug.md)
  [`LearnerClassifDebug`](https://mlr3.mlr-org.com/reference/mlr_learners_classif.debug.md)
  : Classification Learner for Debugging
- [`mlr_learners_classif.featureless`](https://mlr3.mlr-org.com/reference/mlr_learners_classif.featureless.md)
  [`LearnerClassifFeatureless`](https://mlr3.mlr-org.com/reference/mlr_learners_classif.featureless.md)
  : Featureless Classification Learner
- [`mlr_learners_classif.rpart`](https://mlr3.mlr-org.com/reference/mlr_learners_classif.rpart.md)
  [`LearnerClassifRpart`](https://mlr3.mlr-org.com/reference/mlr_learners_classif.rpart.md)
  : Classification Tree Learner
- [`mlr_learners_regr.debug`](https://mlr3.mlr-org.com/reference/mlr_learners_regr.debug.md)
  [`LearnerRegrDebug`](https://mlr3.mlr-org.com/reference/mlr_learners_regr.debug.md)
  : Regression Learner for Debugging
- [`mlr_learners_regr.featureless`](https://mlr3.mlr-org.com/reference/mlr_learners_regr.featureless.md)
  [`LearnerRegrFeatureless`](https://mlr3.mlr-org.com/reference/mlr_learners_regr.featureless.md)
  : Featureless Regression Learner
- [`mlr_learners_regr.rpart`](https://mlr3.mlr-org.com/reference/mlr_learners_regr.rpart.md)
  [`LearnerRegrRpart`](https://mlr3.mlr-org.com/reference/mlr_learners_regr.rpart.md)
  : Regression Tree Learner
- [`as_learner()`](https://mlr3.mlr-org.com/reference/as_learner.md)
  [`as_learners()`](https://mlr3.mlr-org.com/reference/as_learner.md) :
  Convert to a Learner
- [`HotstartStack`](https://mlr3.mlr-org.com/reference/HotstartStack.md)
  : Stack for Hot Start Learners
- [`default_fallback()`](https://mlr3.mlr-org.com/reference/default_fallback.md)
  : Create a Fallback Learner

## Measures

- [`mlr_measures`](https://mlr3.mlr-org.com/reference/mlr_measures.md) :
  Dictionary of Performance Measures
- [`mlr_measures_aic`](https://mlr3.mlr-org.com/reference/mlr_measures_aic.md)
  [`MeasureAIC`](https://mlr3.mlr-org.com/reference/mlr_measures_aic.md)
  : Akaike Information Criterion Measure
- [`mlr_measures_bic`](https://mlr3.mlr-org.com/reference/mlr_measures_bic.md)
  [`MeasureBIC`](https://mlr3.mlr-org.com/reference/mlr_measures_bic.md)
  : Bayesian Information Criterion Measure
- [`mlr_measures_classif.acc`](https://mlr3.mlr-org.com/reference/mlr_measures_classif.acc.md)
  : Classification Accuracy
- [`mlr_measures_classif.auc`](https://mlr3.mlr-org.com/reference/mlr_measures_classif.auc.md)
  : Area Under the ROC Curve
- [`mlr_measures_classif.bacc`](https://mlr3.mlr-org.com/reference/mlr_measures_classif.bacc.md)
  : Balanced Accuracy
- [`mlr_measures_classif.bbrier`](https://mlr3.mlr-org.com/reference/mlr_measures_classif.bbrier.md)
  : Binary Brier Score
- [`mlr_measures_classif.ce`](https://mlr3.mlr-org.com/reference/mlr_measures_classif.ce.md)
  : Classification Error
- [`mlr_measures_classif.costs`](https://mlr3.mlr-org.com/reference/mlr_measures_classif.costs.md)
  [`MeasureClassifCosts`](https://mlr3.mlr-org.com/reference/mlr_measures_classif.costs.md)
  : Cost-sensitive Classification Measure
- [`mlr_measures_classif.dor`](https://mlr3.mlr-org.com/reference/mlr_measures_classif.dor.md)
  : Diagnostic Odds Ratio
- [`mlr_measures_classif.fbeta`](https://mlr3.mlr-org.com/reference/mlr_measures_classif.fbeta.md)
  : F-beta Score
- [`mlr_measures_classif.fdr`](https://mlr3.mlr-org.com/reference/mlr_measures_classif.fdr.md)
  : False Discovery Rate
- [`mlr_measures_classif.fn`](https://mlr3.mlr-org.com/reference/mlr_measures_classif.fn.md)
  : False Negatives
- [`mlr_measures_classif.fnr`](https://mlr3.mlr-org.com/reference/mlr_measures_classif.fnr.md)
  : False Negative Rate
- [`mlr_measures_classif.fomr`](https://mlr3.mlr-org.com/reference/mlr_measures_classif.fomr.md)
  : False Omission Rate
- [`mlr_measures_classif.fp`](https://mlr3.mlr-org.com/reference/mlr_measures_classif.fp.md)
  : False Positives
- [`mlr_measures_classif.fpr`](https://mlr3.mlr-org.com/reference/mlr_measures_classif.fpr.md)
  : False Positive Rate
- [`mlr_measures_classif.logloss`](https://mlr3.mlr-org.com/reference/mlr_measures_classif.logloss.md)
  : Log Loss
- [`mlr_measures_classif.mauc_au1p`](https://mlr3.mlr-org.com/reference/mlr_measures_classif.mauc_au1p.md)
  : Multiclass AUC Scores
- [`mlr_measures_classif.mauc_au1u`](https://mlr3.mlr-org.com/reference/mlr_measures_classif.mauc_au1u.md)
  : Multiclass AUC Scores
- [`mlr_measures_classif.mauc_aunp`](https://mlr3.mlr-org.com/reference/mlr_measures_classif.mauc_aunp.md)
  : Multiclass AUC Scores
- [`mlr_measures_classif.mauc_aunu`](https://mlr3.mlr-org.com/reference/mlr_measures_classif.mauc_aunu.md)
  : Multiclass AUC Scores
- [`mlr_measures_classif.mauc_mu`](https://mlr3.mlr-org.com/reference/mlr_measures_classif.mauc_mu.md)
  : Multiclass AUC Scores
- [`mlr_measures_classif.mbrier`](https://mlr3.mlr-org.com/reference/mlr_measures_classif.mbrier.md)
  : Multiclass Brier Score
- [`mlr_measures_classif.mcc`](https://mlr3.mlr-org.com/reference/mlr_measures_classif.mcc.md)
  : Matthews Correlation Coefficient
- [`mlr_measures_classif.npv`](https://mlr3.mlr-org.com/reference/mlr_measures_classif.npv.md)
  : Negative Predictive Value
- [`mlr_measures_classif.ppv`](https://mlr3.mlr-org.com/reference/mlr_measures_classif.ppv.md)
  : Positive Predictive Value
- [`mlr_measures_classif.prauc`](https://mlr3.mlr-org.com/reference/mlr_measures_classif.prauc.md)
  : Area Under the Precision-Recall Curve
- [`mlr_measures_classif.precision`](https://mlr3.mlr-org.com/reference/mlr_measures_classif.precision.md)
  : Positive Predictive Value
- [`mlr_measures_classif.recall`](https://mlr3.mlr-org.com/reference/mlr_measures_classif.recall.md)
  : True Positive Rate
- [`mlr_measures_classif.sensitivity`](https://mlr3.mlr-org.com/reference/mlr_measures_classif.sensitivity.md)
  : True Positive Rate
- [`mlr_measures_classif.specificity`](https://mlr3.mlr-org.com/reference/mlr_measures_classif.specificity.md)
  : True Negative Rate
- [`mlr_measures_classif.tn`](https://mlr3.mlr-org.com/reference/mlr_measures_classif.tn.md)
  : True Negatives
- [`mlr_measures_classif.tnr`](https://mlr3.mlr-org.com/reference/mlr_measures_classif.tnr.md)
  : True Negative Rate
- [`mlr_measures_classif.tp`](https://mlr3.mlr-org.com/reference/mlr_measures_classif.tp.md)
  : True Positives
- [`mlr_measures_classif.tpr`](https://mlr3.mlr-org.com/reference/mlr_measures_classif.tpr.md)
  : True Positive Rate
- [`mlr_measures_debug_classif`](https://mlr3.mlr-org.com/reference/mlr_measures_debug_classif.md)
  [`MeasureDebugClassif`](https://mlr3.mlr-org.com/reference/mlr_measures_debug_classif.md)
  : Debug Measure for Classification
- [`mlr_measures_elapsed_time`](https://mlr3.mlr-org.com/reference/mlr_measures_elapsed_time.md)
  [`MeasureElapsedTime`](https://mlr3.mlr-org.com/reference/mlr_measures_elapsed_time.md)
  [`mlr_measures_time_train`](https://mlr3.mlr-org.com/reference/mlr_measures_elapsed_time.md)
  [`mlr_measures_time_predict`](https://mlr3.mlr-org.com/reference/mlr_measures_elapsed_time.md)
  [`mlr_measures_time_both`](https://mlr3.mlr-org.com/reference/mlr_measures_elapsed_time.md)
  : Elapsed Time Measure
- [`mlr_measures_internal_valid_score`](https://mlr3.mlr-org.com/reference/mlr_measures_internal_valid_score.md)
  [`MeasureInternalValidScore`](https://mlr3.mlr-org.com/reference/mlr_measures_internal_valid_score.md)
  : Measure Internal Validation Score
- [`mlr_measures_oob_error`](https://mlr3.mlr-org.com/reference/mlr_measures_oob_error.md)
  [`MeasureOOBError`](https://mlr3.mlr-org.com/reference/mlr_measures_oob_error.md)
  : Out-of-bag Error Measure
- [`mlr_measures_regr.bias`](https://mlr3.mlr-org.com/reference/mlr_measures_regr.bias.md)
  : Bias
- [`mlr_measures_regr.ktau`](https://mlr3.mlr-org.com/reference/mlr_measures_regr.ktau.md)
  : Kendall's tau
- [`mlr_measures_regr.mae`](https://mlr3.mlr-org.com/reference/mlr_measures_regr.mae.md)
  : Mean Absolute Error
- [`mlr_measures_regr.mape`](https://mlr3.mlr-org.com/reference/mlr_measures_regr.mape.md)
  : Mean Absolute Percent Error
- [`mlr_measures_regr.maxae`](https://mlr3.mlr-org.com/reference/mlr_measures_regr.maxae.md)
  : Max Absolute Error
- [`mlr_measures_regr.medae`](https://mlr3.mlr-org.com/reference/mlr_measures_regr.medae.md)
  : Median Absolute Error
- [`mlr_measures_regr.medse`](https://mlr3.mlr-org.com/reference/mlr_measures_regr.medse.md)
  : Median Squared Error
- [`mlr_measures_regr.mse`](https://mlr3.mlr-org.com/reference/mlr_measures_regr.mse.md)
  : Mean Squared Error
- [`mlr_measures_regr.msle`](https://mlr3.mlr-org.com/reference/mlr_measures_regr.msle.md)
  : Mean Squared Log Error
- [`mlr_measures_regr.pbias`](https://mlr3.mlr-org.com/reference/mlr_measures_regr.pbias.md)
  : Percent Bias
- [`mlr_measures_regr.pinball`](https://mlr3.mlr-org.com/reference/mlr_measures_regr.pinball.md)
  [`MeasureRegrPinball`](https://mlr3.mlr-org.com/reference/mlr_measures_regr.pinball.md)
  : Average Pinball Loss
- [`mlr_measures_regr.rmse`](https://mlr3.mlr-org.com/reference/mlr_measures_regr.rmse.md)
  : Root Mean Squared Error
- [`mlr_measures_regr.rmsle`](https://mlr3.mlr-org.com/reference/mlr_measures_regr.rmsle.md)
  : Root Mean Squared Log Error
- [`mlr_measures_regr.rqr`](https://mlr3.mlr-org.com/reference/mlr_measures_regr.rqr.md)
  [`MeasureRegrRQR`](https://mlr3.mlr-org.com/reference/mlr_measures_regr.rqr.md)
  : R-Squared for Quantile Regression
- [`mlr_measures_regr.rsq`](https://mlr3.mlr-org.com/reference/mlr_measures_regr.rsq.md)
  [`MeasureRegrRSQ`](https://mlr3.mlr-org.com/reference/mlr_measures_regr.rsq.md)
  : R-Squared
- [`mlr_measures_regr.sae`](https://mlr3.mlr-org.com/reference/mlr_measures_regr.sae.md)
  : Sum of Absolute Errors
- [`mlr_measures_regr.smape`](https://mlr3.mlr-org.com/reference/mlr_measures_regr.smape.md)
  : Symmetric Mean Absolute Percent Error
- [`mlr_measures_regr.srho`](https://mlr3.mlr-org.com/reference/mlr_measures_regr.srho.md)
  : Spearman's rho
- [`mlr_measures_regr.sse`](https://mlr3.mlr-org.com/reference/mlr_measures_regr.sse.md)
  : Sum of Squared Errors
- [`mlr_measures_selected_features`](https://mlr3.mlr-org.com/reference/mlr_measures_selected_features.md)
  [`MeasureSelectedFeatures`](https://mlr3.mlr-org.com/reference/mlr_measures_selected_features.md)
  : Selected Features Measure
- [`mlr_measures_sim.jaccard`](https://mlr3.mlr-org.com/reference/mlr_measures_sim.jaccard.md)
  : Jaccard Similarity Index
- [`mlr_measures_sim.phi`](https://mlr3.mlr-org.com/reference/mlr_measures_sim.phi.md)
  : Phi Coefficient Similarity
- [`default_measures()`](https://mlr3.mlr-org.com/reference/default_measures.md)
  : Get the Default Measure
- [`as_measure()`](https://mlr3.mlr-org.com/reference/as_measure.md)
  [`as_measures()`](https://mlr3.mlr-org.com/reference/as_measure.md) :
  Convert to a Measure
- [`score_roc_measures()`](https://mlr3.mlr-org.com/reference/score_roc_measures.md)
  : Calculate ROC Measures
- [`print(`*`<roc_measures>`*`)`](https://mlr3.mlr-org.com/reference/print.roc_measures.md)
  : Print ROC Measures

## Resampling Strategies

- [`mlr_resamplings`](https://mlr3.mlr-org.com/reference/mlr_resamplings.md)
  : Dictionary of Resampling Strategies
- [`mlr_resamplings_bootstrap`](https://mlr3.mlr-org.com/reference/mlr_resamplings_bootstrap.md)
  [`ResamplingBootstrap`](https://mlr3.mlr-org.com/reference/mlr_resamplings_bootstrap.md)
  : Bootstrap Resampling
- [`mlr_resamplings_custom`](https://mlr3.mlr-org.com/reference/mlr_resamplings_custom.md)
  [`ResamplingCustom`](https://mlr3.mlr-org.com/reference/mlr_resamplings_custom.md)
  : Custom Resampling
- [`mlr_resamplings_custom_cv`](https://mlr3.mlr-org.com/reference/mlr_resamplings_custom_cv.md)
  [`ResamplingCustomCV`](https://mlr3.mlr-org.com/reference/mlr_resamplings_custom_cv.md)
  : Custom Cross-Validation
- [`mlr_resamplings_cv`](https://mlr3.mlr-org.com/reference/mlr_resamplings_cv.md)
  [`ResamplingCV`](https://mlr3.mlr-org.com/reference/mlr_resamplings_cv.md)
  : Cross-Validation Resampling
- [`mlr_resamplings_holdout`](https://mlr3.mlr-org.com/reference/mlr_resamplings_holdout.md)
  [`ResamplingHoldout`](https://mlr3.mlr-org.com/reference/mlr_resamplings_holdout.md)
  : Holdout Resampling
- [`mlr_resamplings_insample`](https://mlr3.mlr-org.com/reference/mlr_resamplings_insample.md)
  [`ResamplingInsample`](https://mlr3.mlr-org.com/reference/mlr_resamplings_insample.md)
  : Insample Resampling
- [`mlr_resamplings_loo`](https://mlr3.mlr-org.com/reference/mlr_resamplings_loo.md)
  [`ResamplingLOO`](https://mlr3.mlr-org.com/reference/mlr_resamplings_loo.md)
  : Leave-One-Out Cross-Validation
- [`mlr_resamplings_repeated_cv`](https://mlr3.mlr-org.com/reference/mlr_resamplings_repeated_cv.md)
  [`ResamplingRepeatedCV`](https://mlr3.mlr-org.com/reference/mlr_resamplings_repeated_cv.md)
  : Repeated Cross-Validation Resampling
- [`mlr_resamplings_subsampling`](https://mlr3.mlr-org.com/reference/mlr_resamplings_subsampling.md)
  [`ResamplingSubsampling`](https://mlr3.mlr-org.com/reference/mlr_resamplings_subsampling.md)
  : Subsampling Resampling
- [`as_resampling()`](https://mlr3.mlr-org.com/reference/as_resampling.md)
  [`as_resamplings()`](https://mlr3.mlr-org.com/reference/as_resampling.md)
  : Convert to a Resampling

## Resample

- [`resample()`](https://mlr3.mlr-org.com/reference/resample.md) :
  Resample a Learner on a Task

- [`partition()`](https://mlr3.mlr-org.com/reference/partition.md) :
  Manually Partition into Training, Test and Validation Set

- [`ResampleResult`](https://mlr3.mlr-org.com/reference/ResampleResult.md)
  :

  Container for Results of
  [`resample()`](https://mlr3.mlr-org.com/reference/resample.md)

- [`as_result_data()`](https://mlr3.mlr-org.com/reference/as_result_data.md)
  : Convert to ResultData

- [`as_resample_result()`](https://mlr3.mlr-org.com/reference/as_resample_result.md)
  : Convert to ResampleResult

## Benchmarking

- [`benchmark()`](https://mlr3.mlr-org.com/reference/benchmark.md) :
  Benchmark Multiple Learners on Multiple Tasks
- [`benchmark_grid()`](https://mlr3.mlr-org.com/reference/benchmark_grid.md)
  : Generate a Benchmark Grid Design
- [`BenchmarkResult`](https://mlr3.mlr-org.com/reference/BenchmarkResult.md)
  : Container for Benchmarking Results
- [`as_result_data()`](https://mlr3.mlr-org.com/reference/as_result_data.md)
  : Convert to ResultData
- [`as_benchmark_result()`](https://mlr3.mlr-org.com/reference/as_benchmark_result.md)
  : Convert to BenchmarkResult
- [`uhashes()`](https://mlr3.mlr-org.com/reference/uhash.md)
  [`uhash()`](https://mlr3.mlr-org.com/reference/uhash.md) : Obtain
  specific uhashes from a BenchmarkResult

## Converters

- [`as_benchmark_result()`](https://mlr3.mlr-org.com/reference/as_benchmark_result.md)
  : Convert to BenchmarkResult
- [`as_data_backend()`](https://mlr3.mlr-org.com/reference/as_data_backend.md)
  : Create a Data Backend
- [`as_learner()`](https://mlr3.mlr-org.com/reference/as_learner.md)
  [`as_learners()`](https://mlr3.mlr-org.com/reference/as_learner.md) :
  Convert to a Learner
- [`as_measure()`](https://mlr3.mlr-org.com/reference/as_measure.md)
  [`as_measures()`](https://mlr3.mlr-org.com/reference/as_measure.md) :
  Convert to a Measure
- [`as_prediction()`](https://mlr3.mlr-org.com/reference/as_prediction.md)
  [`as_predictions()`](https://mlr3.mlr-org.com/reference/as_prediction.md)
  : Convert to a Prediction
- [`as_prediction_classif()`](https://mlr3.mlr-org.com/reference/as_prediction_classif.md)
  : Convert to a Classification Prediction
- [`as_prediction_data()`](https://mlr3.mlr-org.com/reference/as_prediction_data.md)
  : PredictionData
- [`as_prediction_regr()`](https://mlr3.mlr-org.com/reference/as_prediction_regr.md)
  : Convert to a Regression Prediction
- [`as_resample_result()`](https://mlr3.mlr-org.com/reference/as_resample_result.md)
  : Convert to ResampleResult
- [`as_resampling()`](https://mlr3.mlr-org.com/reference/as_resampling.md)
  [`as_resamplings()`](https://mlr3.mlr-org.com/reference/as_resampling.md)
  : Convert to a Resampling
- [`as_result_data()`](https://mlr3.mlr-org.com/reference/as_result_data.md)
  : Convert to ResultData
- [`as_task()`](https://mlr3.mlr-org.com/reference/as_task.md)
  [`as_tasks()`](https://mlr3.mlr-org.com/reference/as_task.md) :
  Convert to a Task
- [`as_task_classif()`](https://mlr3.mlr-org.com/reference/as_task_classif.md)
  : Convert to a Classification Task
- [`as_task_regr()`](https://mlr3.mlr-org.com/reference/as_task_regr.md)
  : Convert to a Regression Task
- [`as_task_unsupervised()`](https://mlr3.mlr-org.com/reference/as_task_unsupervised.md)
  [`as_tasks_unsupervised()`](https://mlr3.mlr-org.com/reference/as_task_unsupervised.md)
  : Convert to an Unsupervised Task

## Syntactic Sugar and Reflections

- [`install_pkgs()`](https://mlr3.mlr-org.com/reference/install_pkgs.md)
  [`extract_pkgs()`](https://mlr3.mlr-org.com/reference/install_pkgs.md)
  : Install (Missing) Packages
- [`tsk()`](https://mlr3.mlr-org.com/reference/mlr_sugar.md)
  [`tsks()`](https://mlr3.mlr-org.com/reference/mlr_sugar.md)
  [`tgen()`](https://mlr3.mlr-org.com/reference/mlr_sugar.md)
  [`tgens()`](https://mlr3.mlr-org.com/reference/mlr_sugar.md)
  [`lrn()`](https://mlr3.mlr-org.com/reference/mlr_sugar.md)
  [`lrns()`](https://mlr3.mlr-org.com/reference/mlr_sugar.md)
  [`rsmp()`](https://mlr3.mlr-org.com/reference/mlr_sugar.md)
  [`rsmps()`](https://mlr3.mlr-org.com/reference/mlr_sugar.md)
  [`msr()`](https://mlr3.mlr-org.com/reference/mlr_sugar.md)
  [`msrs()`](https://mlr3.mlr-org.com/reference/mlr_sugar.md)
  [`set_validate()`](https://mlr3.mlr-org.com/reference/mlr_sugar.md) :
  Syntactic Sugar for Object Construction
- [`mlr_reflections`](https://mlr3.mlr-org.com/reference/mlr_reflections.md)
  : Reflections for mlr3
- [`set_threads()`](https://mlr3.mlr-org.com/reference/set_threads.md) :
  Set the Number of Threads

## Callbacks

- [`CallbackResample`](https://mlr3.mlr-org.com/reference/CallbackResample.md)
  : Resample Callback
- [`ContextResample`](https://mlr3.mlr-org.com/reference/ContextResample.md)
  : Resample Context
- [`callback_resample()`](https://mlr3.mlr-org.com/reference/callback_resample.md)
  : Create Evaluation Callback
- [`assert_resample_callback()`](https://mlr3.mlr-org.com/reference/assert_resample_callback.md)
  [`assert_resample_callbacks()`](https://mlr3.mlr-org.com/reference/assert_resample_callback.md)
  : Assertions for Callbacks
- [`mlr3.model_extractor`](https://mlr3.mlr-org.com/reference/mlr3.model_extractor.md)
  : Model Extractor Callback
- [`mlr3.holdout_task`](https://mlr3.mlr-org.com/reference/mlr3.holdout_task.md)
  : Callback Holdout Task

## Internal Objects and Functions

- [`learner_unmarshal()`](https://mlr3.mlr-org.com/reference/marshaling.md)
  [`learner_marshal()`](https://mlr3.mlr-org.com/reference/marshaling.md)
  [`learner_marshaled()`](https://mlr3.mlr-org.com/reference/marshaling.md)
  [`marshal_model()`](https://mlr3.mlr-org.com/reference/marshaling.md)
  [`unmarshal_model()`](https://mlr3.mlr-org.com/reference/marshaling.md)
  [`is_marshaled_model()`](https://mlr3.mlr-org.com/reference/marshaling.md)
  : (Un)marshal a Learner
- [`MeasureSimilarity`](https://mlr3.mlr-org.com/reference/MeasureSimilarity.md)
  : Similarity Measure
- [`assert_backend()`](https://mlr3.mlr-org.com/reference/mlr_assertions.md)
  [`assert_task()`](https://mlr3.mlr-org.com/reference/mlr_assertions.md)
  [`assert_tasks()`](https://mlr3.mlr-org.com/reference/mlr_assertions.md)
  [`assert_learner()`](https://mlr3.mlr-org.com/reference/mlr_assertions.md)
  [`assert_learners()`](https://mlr3.mlr-org.com/reference/mlr_assertions.md)
  [`assert_learnable()`](https://mlr3.mlr-org.com/reference/mlr_assertions.md)
  [`assert_predictable()`](https://mlr3.mlr-org.com/reference/mlr_assertions.md)
  [`assert_measure()`](https://mlr3.mlr-org.com/reference/mlr_assertions.md)
  [`assert_scorable()`](https://mlr3.mlr-org.com/reference/mlr_assertions.md)
  [`assert_measures()`](https://mlr3.mlr-org.com/reference/mlr_assertions.md)
  [`assert_resampling()`](https://mlr3.mlr-org.com/reference/mlr_assertions.md)
  [`assert_resamplings()`](https://mlr3.mlr-org.com/reference/mlr_assertions.md)
  [`assert_prediction()`](https://mlr3.mlr-org.com/reference/mlr_assertions.md)
  [`assert_resample_result()`](https://mlr3.mlr-org.com/reference/mlr_assertions.md)
  [`assert_benchmark_result()`](https://mlr3.mlr-org.com/reference/mlr_assertions.md)
  [`assert_row_ids()`](https://mlr3.mlr-org.com/reference/mlr_assertions.md)
  [`assert_has_backend()`](https://mlr3.mlr-org.com/reference/mlr_assertions.md)
  [`assert_quantiles()`](https://mlr3.mlr-org.com/reference/mlr_assertions.md)
  [`assert_validate()`](https://mlr3.mlr-org.com/reference/mlr_assertions.md)
  : Assertion for mlr3 Objects
- [`ResultData`](https://mlr3.mlr-org.com/reference/ResultData.md) :
  ResultData
- [`create_empty_prediction_data()`](https://mlr3.mlr-org.com/reference/PredictionData.md)
  [`check_prediction_data()`](https://mlr3.mlr-org.com/reference/PredictionData.md)
  [`is_missing_prediction_data()`](https://mlr3.mlr-org.com/reference/PredictionData.md)
  [`filter_prediction_data()`](https://mlr3.mlr-org.com/reference/PredictionData.md)
  [`c(`*`<PredictionDataClassif>`*`)`](https://mlr3.mlr-org.com/reference/PredictionData.md)
  [`c(`*`<PredictionDataRegr>`*`)`](https://mlr3.mlr-org.com/reference/PredictionData.md)
  : Convert to PredictionData
- [`predict(`*`<Learner>`*`)`](https://mlr3.mlr-org.com/reference/predict.Learner.md)
  : Predict Method for Learners
- [`mlr_test_helpers`](https://mlr3.mlr-org.com/reference/mlr_test_helpers.md)
  : Documentation of mlr3 test helpers

## Package

- [`mlr3`](https://mlr3.mlr-org.com/reference/mlr3-package.md)
  [`mlr3-package`](https://mlr3.mlr-org.com/reference/mlr3-package.md) :
  mlr3: Machine Learning in R - Next Generation