Changelog
Source:NEWS.md
mlr3 0.21.0
CRAN release: 2024-09-24
- BREAKING CHANGE: Deprecated
data_format
anddata_formats
forLearner
,Task
, andDataBackend
classes. - feat: The
partition()
function creates training, test and validation sets now. - perf: Optimize the runtime of fixing factor levels.
- perf: Optimize the runtime of setting row roles.
- perf: Optimize the runtime of marshalling.
- perf: Optimize the runtime of
Task$col_info
. - fix: column info is now checked for compatibility during
Learner$predict
(#943). - BREAKING CHANGE: The predict time of the learner now stores the cumulative duration for all predict sets (#992).
- feat:
$internal_valid_task
can now be set to aninteger
vector. - feat: Measures can now have an empty
$predict_sets
(#1094). This is relevant for measures that only extract information from the model of a learner (such as internal validation scores or AIC / BIC) - BREAKING CHANGE: Deprecated the
$divide()
method - fix:
Task$cbind()
now works with non-standard primary keys fordata.frames
(#961). - fix: Triggering of fallback learner now has log-level
"info"
instead of"debug"
(#972). - feat: Added new measure
regr.pinball
here and in mlr3measures. - feat: Added new measure
mu_auc
here and in mlr3measures. - feat: Add option to calculate the mean of the true values on the train set in
msr("regr.rsq")
. - feat: Default fallback learner is set when encapsulation is activated.
- feat: Learners
classif.debug
andregr.debug
have new methods$importance()
and$selected_features()
for testing, also in downstream packages. - feat: Create default fallback learner with
default_fallback()
. - feat: Check column roles when using
$set_col_roles()
and$col_roles
. - fix: Add predict set to learner hash.
- BREAKING CHANGE: Encapsulation and the fallback learner are now set with the
$encapsulate(method, fallback)
method. The$fallback
field is read-only now and the encapsulate status can be retrieved from the$encapsulation
field.
mlr3 0.20.2
CRAN release: 2024-07-29
- refactor: Move RhpcBLASctl to suggest.
- feat: Added resampling property
"primary_iters"
- feat: Added possibility to access observation-wise losses via function
$obs_loss
. This is possible forPrediction
,ResampleResult
andBenchmarkResult
. - feat:
Measure
s now also return a vector of numerics.
mlr3 0.20.1
CRAN release: 2024-07-22
- feat: Add multiclass Matthews correlation coefficient
msr("classif.mcc")
.
mlr3 0.19.0
CRAN release: 2024-04-24
- Added support for
"marshal"
property, which allows learners to process models so they can be serialized. This happens automatically duringresample()
andbenchmark()
. - Encapsulation methods use the same RNG state now.
- Fix missing values in
default_values.Learner()
function. - Encapsulated error messages are now printed with the
lgr
package.
mlr3 0.18.0
CRAN release: 2024-03-05
- Prepare compatibility with new paradox version.
- Dictionary conversion of
mlr_learners
respects prototype arguments recently added in mlr3misc. - Skip unnecessary clone of learner’s state in
resample()
.
mlr3 0.17.1
CRAN release: 2023-12-21
- Remove
data_prototype
when resampling fromlearner$state
to reduce memory consumption. - Reduce number of threads used by
data.table
and BLAS to 1 when runningresample()
orbenchmark()
in parallel. - Optimize runtime of
resample()
andbenchmark()
by reducing the number of hashing operations.
mlr3 0.17.0
CRAN release: 2023-11-17
- Learners cannot be added to the
HotstartStack
anymore when the model is missing. - Learners bellow the
hotstart_threshold
are not added to theHotstartStack
anymore. - The
learner$state$train_time
in hotstarted learners is now only the time of the last training. - Added debug messages to the hotstart stack.
- Fixed bug where the
HotstartStack
did not work with column roles set in the task. - The
design
ofbenchmark()
can now include parameter settings. - Speed up resampling by removing unnecessary calls to
packageVersion()
. - Fix boston housing data set.
- Export generic function
col_info
to allow adding new methods for backends. - Task printer includes row roles now.
- Add
"mlr3.exec_chunk_bins"
option to split the resampling iterations into a number of bins.
mlr3 0.16.1
CRAN release: 2023-06-17
- Function
data.table()
is now re-exported. - Fixed a test which randomly failed.
- Improved documentation.
- Add encapsulation mode
"try"
, which works similar to"none"
but captures errors
mlr3 0.16.0
CRAN release: 2023-05-05
- Added argument
paired
tobenchmark_grid()
function, which can be used to create a benchmark design, where resamplings have been instantiated on tasks. - Added S3 method for
ResultData
foras_resample_result()
converter. - Added S3 method for
list
foras_resample_result()
converter. - The featureless classification learner now returns proper probabilities (#918).
mlr3 0.15.0
CRAN release: 2023-03-17
- Many returned tables are now assigned a class for a
print
method to make the output more readable. - Fixed some typos
mlr3 0.14.1
CRAN release: 2022-11-02
- Removed dependency on package
distr6
. - Fixed reassembling of
GraphLearner
. - Fixed bug where the measured elapsed time was 0: https://stackoverflow.com/questions/73797845/mlr3-benchmarking-with-elapsed-time-measure
- Fixed
as_prediction_classif()
fordata.frame()
input (#872). - Improved the error message when predict type of fallback learner does not match the predict type of the learner (mlr-org/mlr3extralearners#241).
- The test set is now available to the
Learner
during train for early stopping.
mlr3 0.14.0
CRAN release: 2022-08-11
- Added multiclass measures:
mauc_aunu
,mauc_aunp
,mauc_au1u
,mauc_au1p
. - Measure
classif.costs
does not require aTask
anymore. - New converter:
as_task_unsupervised()
- Refactored the task types in
mlr_reflections
.
mlr3 0.13.4
CRAN release: 2022-07-21
- Added new options for parallelization (
"mlr3.exec_random"
and"mlr3.exec_chunk_size"
). These options are passed down to the respective map functions in packagefuture.apply
. - Fixed runtime measures depending on specific predict types (#832).
- Added
head()
andtail()
methods forTask
. - Improved printing of multiple objects.
mlr3 0.13.3
CRAN release: 2022-03-01
- Most objects now have a new (optional) field
label
, i.e.Task
,TaskGenerator
,Learner
,Resampling
, andMeasure
. -
as.data.table()
methods for objects of classDictonary
have been extended with additional columns. -
as_task_classif.formula()
andas_task_regr.formula()
now remove additional atrributes attached to the data which caused some some learners to break. - Packages are now loaded prior to calling the
$train()
and$predict()
methods of aLearner
. This ensures that package loading errors are properly propagated and not affected by encapsulation (#771).
mlr3 0.13.2
CRAN release: 2022-02-14
- Setting a fallback learner for a learner with encapsulation in its default settings now automatically sets encapsulation to
"evaluate"
(#763). -
as_task_classif()
andas_task_regr()
now support the construction of tasks using the formula interface, e.g.as_task_regr(mpg ~ ., data = mtcars)
(#761). - Added
default_values()
function to extract parameter default values fromLearner
objects. - The row role
"validation"
has been renamed to"holdout"
. In the next release,mlr3
will start switching to the now more common terms"train"
/"validation"
instead of"train"
/"test"
for the sets created during resampling.
mlr3 0.13.1
CRAN release: 2022-01-19
- Improved performance for many operations on
ResampleResult
andBenchmarkResult
. -
resample()
andbenchmark()
got a new argumentclone
to control which objects to clone before performing computations. - Tasks are checked for infinite values during the conversion from
data.frame
toTask
inas_task_classif()
andas_task_regr()
. A warning is signaled if any column contains infinite values.
mlr3 0.13.0
CRAN release: 2021-11-16
- Learners which are capable of resuming/continuing (e.g., learner
(classif|regr|surv).xgboost
with hyperparameternrounds
updated) can now optionally store a stack of trained learners to be used to hotstart their training. Note that this feature is still somewhat experimental. SeeHotstartStack
and #719. - New measures to score similarity of selected feature sets:
sim.jaccard
(Jaccard Index) andsim.phi
(Phi coefficient) (#690). -
predict_newdata()
now also supportsDataBackend
as input. - New function
install_pkgs()
to install required packages. This generic works for all objects with apackages
field as well asResampleResult
andBenchmarkResult
(#728). - New learner
regr.debug
for debugging. - New
Task
method$set_levels()
to control how data with factor columns is returned, independent of the usedDataBackend
. - Measures now return
NA
if prerequisite are not met (#699). This allows to conveniently score your experiments with multiple measures having different requirements. - Feature names may no longer contain the special character
%
.
mlr3 0.12.0
CRAN release: 2021-08-05
- New method to assign labels to columns in tasks:
Task$label()
. These will be used in visualizations in the future. - New method to add stratification variables:
Task$add_strata()
. - New helper function
partition()
to split a task into a training and test set. - New standardized getter
loglik()
for classLearner
. - New measures
"aic"
and"bic"
to compute the Akaike Information Criterion or the Bayesian Information Criterion, respectively. - New Resampling method:
ResamplingCustomCV
. Creates a custom resampling split based on the levels of a user-provided factor variable. - New argument
encapsulate
forresample()
andbenchmark()
to conveniently enable encapsulation and also set the fallback learner to the featureless learner. This is simply for convenience, configuring each learner individually is still possible and allows a more fine-grained control (#634, #642). - New field
parallel_predict
forLearner
to enable parallel predictions via the future backend. This currently is only enabled while calling the$predict()
or$predict_newdata
methods and is disabled duringresample()
andbenchmark()
where you have other means to parallelize. - Deprecated public (and already documented as internal) field
$data
inResampleResult
andBenchmarkResult
to simplify the API and avoid confusion. The converteras.data.table()
can be used instead to access the internal data. - Measures now have formal hyperparameters. A popular example where this is required is the F1 score, now implemented with customizable
beta
. - Changed default of argument
ordered
inTask$data()
fromTRUE
toFALSE
. - Fixed getter
ResamplingRepeatedCV$folds()
(#643). - Fixed hashing of some measures.
- Removed experimental column role
uri
. This role be split up into multiple roles by themlr3keras
package. - Update paramtest to error on extra parameters
mlr3 0.11.0
CRAN release: 2021-03-05
- Added a
as.data.table.Resampling
method. - Renamed column
"row_id"
to"row_ids"
in theas.data.table()
methods forPredictionClassif
andPredictionRegr
(#547). - Added converters
as_prediction_classif()
andas_prediction_regr()
to reverse the operation ofas.data.table.PredictionClassif()
andas.data.table.PredictionRegr()
. - Specifying a weight column during
learner$predict_newdata()
is not mandatory anymore (#563). -
Task$data()
defaults to return only active rows and columns, instead of asserting to only return rows and columns. As a result, the$data()
method can now also be used to query inactive rows and cols from theDataBackend
. - New (experimental) column role
uri
which is intended to point to external resources, e.g. images on the file system. - New helper
set_threads()
to control the number of threads during calls to external packages. All objects will be migrated to have threading disabled in their defaults to avoid conflicting parallelization techniques (#605). - New option
mlr3.debug
: avoid calls tofuture
inresample()
andbenchmark()
to improve the readability of tracebacks. - New experimental option
mlr3.allow_utf8_names
: allow non-ascii characters in column names in tasks.
mlr3 0.10.0
CRAN release: 2021-01-21
- Result containers
ResampleResult
andBenchmarkResult
now optionally remove the DataBackend of the Tasks in order to reduce file size and memory footprint after serialization. To remove the backends from the containers, setstore_backends
toFALSE
inresample()
orbenchmark()
, respectively. Note that this behavior will eventually will be the default for future releases. - Prediction objects generated by
Learner$predict_newdata()
now have row ids starting from 1 instead auto incremented row ids of the training task. -
as.data.table.DictionaryTasks
now returns an additional columnproperties
. - Added flag
conditions
toResampleResult$score()
andBenchmarkResult$score()
to allow to work with failing learners more conveniently.
mlr3 0.9.0
CRAN release: 2020-12-06
- New methods for
Task
:$set_col_roles
and$set_row_roles
as a replacement for the deprecated and less flexible$set_col_role
and$set_row_role
. - Learners can now have a timeout (#556).
- Removed S3 method
friedman.test.BenchmarkResult()
in favor of the newmlr3benchmark
package.
mlr3 0.8.0
CRAN release: 2020-10-21
-
MeasureOOBError
now has set propertyminimize
toTRUE
. - New learner property
"featureless"
to tag learners which can operate on featureless tasks. - Fixed [ResampleResult] ignoring argument
predict_sets
for returned [Prediction] objects. - Compatibility with new version of
lgr
.
mlr3 0.7.0
CRAN release: 2020-10-07
- Updated properties of featureless learners to apply it on all feature types (did not work on POSIXct columns).
- Fixed measures being calculated as
NaN
forBenchmarkResult
for resamplings with a single iteration (#551). - Fixed a bug where a broken heuristic disabled nested parallelization via package
future
(mlr3tuning#270). -
ResampleResult
andBenchmarkResult
now share a common interface to store the experiment results. Manual construction is still possible with helper functionas_result_data()
- Fixed deep cloning of
ResamplingCV
andResamplingRepeatedCV
. - New measure
classif.prauc
(area under precision-recall curve). - Removed dependency on orphaned package
bibtex
.
mlr3 0.6.0
CRAN release: 2020-09-13
- Compact in-memory representation of R6 objects to save space when saving objects via
saveRDS()
orserialize()
. - Objects in containers like
ResampleResult
orBenchmarkResult
are now de-duplicated for an optimized serialization. - Fixed data set
breast_cancer
: all factor features are now correctly stored as ordered factors. - Added a new utility function
convert_task()
.
mlr3 0.5.0
CRAN release: 2020-08-07
- Added classification task
breast_cancer
- Added
ResamplingLOO
for leave-one-out resampling. - Regression now supports predict type
"distr"
using thedistr6
package. - Fixed
ResamplingBootstrap
in combination with grouping (#514). - Fixed plot method of
TaskGeneratorMoons
. - Added hyperparameter
keep_model
to learners"classif.rpart"
and"regr.rpart"
.
mlr3 0.3.0
CRAN release: 2020-06-02
- Package
future.apply
is now imported (instead of suggested). This is necessary to ensure reproducibility: This way exactly the same result is calculated, independent of the parallel backend. - Fixed a bug where prediction on new data for a task with blocking information raised an exception (#496).
- New binding:
Task$order
.
mlr3 0.2.0
CRAN release: 2020-04-17
- Some handy cheat sheets can now be downloaded from the project homepage.
- Added new measures
classif.bbrier
(binary Brier score) andclassif.mbrier
(multi-class Brier score). - Added new Resampling:
ResamplingInsample
. - Added base class for unsupervised tasks:
TaskUnsupervised
.
mlr3 0.1.7
CRAN release: 2020-02-23
Switched to new
roxygen2
documentation format for R6 classes.resample()
andbenchmark()
now support progress bars via the packageprogressr
.Row ids now must be numeric. It was previously allowed to have character row ids, but this lead to confusion and unnecessary code bloat. Row identifiers (e.g., to be used in plots) can still be part of the task, with row role
"name"
.Row names can now be queried with
Task$row_names
.DataBackendMatrix
now supports to store an optional (numeric) dense part.Added new method
$filter()
to filterResampleResult
s to a subset of iterations.Removed deprecated
character()
-> object converters.Empty test sets are now handled separately by learners (#421). An empty prediction object is returned for all learners.
The internal train and predict function of
Learner
now should be implemented as private method: instead of public methodstrain_internal
andpredict_internal
, private methods.train
and.predict
are now encouraged.-
It is now encouraged to move some internal methods from public to private:
-
Learner$train_internal
should now be private method$.train
. -
Learner$predict_internal
should now be private method$.predict
. -
Measure$score_internal
should now be private method$.score
. The public methods will be deprecated in a future release.
-
Removed arguments from the constructor of measures
classif.debug
andclassif.costs
. These can be set directly bymsr()
.
mlr3 0.1.6
CRAN release: 2019-12-19
We have published an article about mlr3 in the Journal of Open Source Software: https://joss.theoj.org/papers/10.21105/joss.01903. See
citation("mlr3")
for the citation info.New method
Learner$reset()
.New method
BenchmarkResult$filter()
.Learners returned by
BenchmarkResult$learners
are reset to encourage the safer alternativeBenchmarkResult$score()
to access trained models.Fix ordering of levels in
PredictionClassif$set_threshold()
(triggered an assertion).
mlr3 0.1.5
CRAN release: 2019-12-10
Switched from package
Metrics
to packagemlr3measures
.Measures can now calculate all scores using micro or macro averaging (#400).
Measures can now be configured to return a customizable performance score (instead of
NA
) in case the score cannot be calculated.Character columns are now treated differently from factor columns. In the long term,
character()
columns are supposed to store text.Fixed a bug triggered by integer grouping variables in
Task
(#396).benchmark_grid()
now accepts instantiated resamplings under certain conditions.
mlr3 0.1.4
CRAN release: 2019-10-28
Task$set_col_roles()
andTask$set_row_roles()
are now deprecated. Instead it is recommended for now to work with the listsTask$col_roles
andTask$row_roles
directly.Learner$predict_newdata()
now works without argumenttask
if the learner has been fitted withLearner$train()
(#375).Names of column roles have been unified (
"weights"
,"label"
,"stratify"
and"groups"
have been renamed).Replaced
MeasureClassifF1
withMeasureClassifFScore
and fixed a bug in the F1 performance calculation (#353). Thanks to @001ben for reporting.Stratification is now controlled via a task column role (was a parameter of class
Resampling
before).Added a S3
predict()
method for classLearner
to increase interoperability with other packages.Many objects now come with a
$help()
which opens the respective manual page.
mlr3 0.1.3
CRAN release: 2019-09-18
It is now possible to predict and score results on the training set or on both training and test set. Learners can be instructed to predict on multiple sets by setting
predict_sets
(default:"test"
). Measures operate on all sets specified in their fieldpredict_sets
(default:"test"
).ResampleResult$prediction
andResampleResult$predictions()
are now methods instead of fields, and allow to extract predictions for different predict sets.ResampleResult$performance()
has been renamed toResampleResult$score()
for consistency.BenchmarkResult$performance()
has been renamed toBenchmarkResult$score()
for consistency.Changed API for (internal) constructors accepting
paradox::ParamSet()
. Instead of passing the initial values separately, the initial values must now be set directly in theParamSet
.
mlr3 0.1.2
CRAN release: 2019-08-25
Deprecated support of automatically creating objects from strings. Instead,
mlr3
provides the following helper functions intended to ease the creation of objects stored in dictionaries:tsk()
,tgen()
,lrn()
,rsmp()
,msr()
.BenchmarkResult
now ensures that the storedResampleResult
s are in a persistent order. Thus,ResampleResult
s can now be addressed by their position instead of their hash.New field
BenchmarkResult$n_resample_results
.New field
BenchmarkResult$hashes
.New method
Task$rename()
.New S3 generic
as_benchmark_result()
.Renamed
Generator
toTaskGenerator
.Removed the control object
mlr_control()
.Removed
ResampleResult$combine()
.Removed
BenchmarkResult$best()
.