Measure Best Validation Score

Returns the selected best internal validation score of the Learner. This is only available for learners that have both the "validation" and the "internal_tuning" property, because tracking a best iteration only makes sense for learners that iterate. Returns NA for unsupported learners, when no validation was done, or when the selected id was not found. The id of this measure is set to the value of select if provided.

While msr("internal_valid_score") reports the validation score of the final model, this measure reports the best validation score observed during training.

Some learners automatically use the best found model for prediction instead of the one from the last iteration. For those the two measures report the same value.

Dictionary

This Measure can be instantiated via the dictionary mlr_measures or with the associated sugar function msr():

mlr_measures$get("best_valid_score")
msr("best_valid_score")

Meta Information

• Task type: “NA”

• Range: \((-\infty, \infty)\)

• Minimize: NA

• Average: macro

• Required Prediction: “NA”

• Required Packages: mlr3

Parameters

Empty ParamSet

See also

• Chapter in the mlr3book: https://mlr3book.mlr-org.com/chapters/chapter2/data_and_basic_modeling.html#sec-eval

• Package mlr3measures for the scoring functions. Dictionary of Measures: mlr_measures as.data.table(mlr_measures) for a table of available Measures in the running session (depending on the loaded packages).

• Extension packages for additional task types:

  • mlr3proba for probabilistic supervised regression and survival analysis.

  • mlr3cluster for unsupervised clustering.

Other Measure: Measure, MeasureClassif, MeasureRegr, MeasureSimilarity, mlr_measures, mlr_measures_aic, mlr_measures_bic, mlr_measures_classif.costs, mlr_measures_debug_classif, mlr_measures_elapsed_time, mlr_measures_internal_valid_score, mlr_measures_oob_error, mlr_measures_regr.pinball, mlr_measures_regr.rqr, mlr_measures_regr.rsq, mlr_measures_selected_features

Super classes

Measure -> MeasureValidScore -> MeasureBestValidScore

Methods

Public methods

• MeasureBestValidScore$new()

• MeasureBestValidScore$clone()

Inherited methods

• Measure$aggregate()
• Measure$format()
• Measure$help()
• Measure$obs_loss()
• Measure$print()
• Measure$score()

---

MeasureBestValidScore$new()

Creates a new instance of this R6 class.

Usage

MeasureBestValidScore$new(select = NULL, minimize = NA)

Arguments

select

(character(1))
Which of the best validation scores to select. Which scores are available depends on the learner and its configuration. By default, the first score is chosen.

minimize

(logical(1))
Whether smaller values are better. Must be set to use for tuning.

---

MeasureBestValidScore$clone()

The objects of this class are cloneable with this method.

Usage

MeasureBestValidScore$clone(deep = FALSE)

Arguments

deep

Whether to make a deep clone.

Examples

rr = resample(tsk("iris"), lrn("classif.debug", validate = 0.3), rsmp("holdout"))
rr$score(msr("best_valid_score", select = "acc"))
#>    task_id    learner_id resampling_id iteration       acc
#>     <char>        <char>        <char>     <int>     <num>
#> 1:    iris classif.debug       holdout         1 0.3666667
#> Hidden columns: task, learner, resampling, prediction_test