Package: cvms 2.0.1

Ludvig Renbo Olsen

cvms: Cross-Validation for Model Selection

Cross-validate one or multiple regression and classification models and get relevant evaluation metrics in a tidy format. Validate the best model on a test set and compare it to a baseline evaluation. Alternatively, evaluate predictions from an external model. Currently supports regression and classification (binary and multiclass). Described in chp. 5 of Jeyaraman, B. P., Olsen, L. R., & Wambugu M. (2019, ISBN: 9781838550134).

Authors:Ludvig Renbo Olsen [aut, cre], Hugh Benjamin Zachariae [aut], Indrajeet Patil [ctb], Daniel Lüdecke [ctb]

cvms_2.0.1.tar.gz
cvms_2.0.1.zip(r-4.7)cvms_2.0.1.zip(r-4.6)cvms_2.0.1.zip(r-4.5)
cvms_2.0.1.tgz(r-4.6-any)cvms_2.0.1.tgz(r-4.5-any)
cvms_2.0.1.tar.gz(r-4.7-any)cvms_2.0.1.tar.gz(r-4.6-any)
cvms_2.0.1.tgz(r-4.6-emscripten)
manual.pdf |manual.html✨
DESCRIPTION |NEWS
card.svg |card.png
cvms/json (API)

# Install 'cvms' in R:
install.packages('cvms', repos = c('https://ludvigolsen.r-universe.dev', 'https://cloud.r-project.org'))

Bug tracker:https://github.com/ludvigolsen/cvms/issues

Datasets:

On CRAN:

Conda:

10.32 score 39 stars 5 packages 496 scripts 1.4k downloads 1 mentions 34 exports 84 dependencies

Last updated from:8c700e0118. Checks:9 OK. Indexed: yes.

TargetResultTimeFilesSyslog
linux-devel-x86_64OK228
source / vignettesOK330
linux-release-x86_64OK244
macos-release-arm64OK177
macos-oldrel-arm64OK134
windows-develOK187
windows-releaseOK196
windows-oldrelOK214
wasm-releaseOK178

Exports:baselinebaseline_binomialbaseline_gaussianbaseline_multinomialbinomial_metricscombine_predictorsconfusion_matrixcross_validatecross_validate_fndynamic_font_color_settingsevaluateevaluate_residualsfontgaussian_metricsmodel_functionsmost_challengingmulticlass_probability_tibblemultinomial_metricsplot_confusion_matrixplot_metric_densitypredict_functionspreprocess_functionsprocess_info_binomialprocess_info_gaussianprocess_info_multinomialreconstruct_formulasselect_definitionsselect_metricssimplify_formulasum_tile_settingssummarize_metricsupdate_hyperparametersvalidatevalidate_fn

Dependencies:backportsbayestestRbootcheckmateclasscliclockcodetoolscpp11data.tabledatawizarddiagramdigestdplyrfarverfuturefuture.applygenericsggplot2globalsgluegowergroupdata2gtablehardhatinsightipredisobandKernSmoothlabelinglatticelavalifecyclelistenvlme4lubridatemagrittrMASSMatrixminqaMuMInnlmenloptrnnetnumbersnumDerivparallellyparameterspillarpkgconfigplyrpROCprodlimprogressrpurrrR6rbibutilsRColorBrewerRcppRcppEigenRdpackrearrrrecipesreformulasrlangrpartS7scalesshapesparsevctrsSQUAREMstringistringrsurvivaltibbletidyrtidyselecttimechangetimeDatetzdbutf8vctrsviridisLitewithr

The available metrics in cvms
Introduction | Gaussian Metrics | Binomial Metrics | Multinomial Metrics | External metrics
Rendered fromavailable_metrics.Rmdusingknitr::rmarkdown

Last update: 2023-06-05
Started: 2020-04-13

Creating a confusion matrix with cvms
Introduction | Attach packages and set seed for reproducibility | Binomial data | Manually creating a two-class confusion matrix | Creating a confusion matrix with evaluate() | Multiclass confusion matrix with confusion_matrix() | Adding sum tiles | Tweaking plot_confusion_matrix()
Rendered fromCreating_a_confusion_matrix.Rmdusingknitr::rmarkdown

Last update: 2023-06-05
Started: 2020-04-13

Cross-validating custom model functions with cvms
Introduction | Attach packages | Prepare data | Predicting the score | Creating a model function | Creating a predict function | Cross-validating the functions | Passing hyperparameters | update_hyperparameters() | Specifying hyperparameters for grid search | Cross-validating hyperparameter combinations | Preprocessing within the cross-validation | Binomial example | Multinomial example
Rendered fromcross_validating_custom_functions.Rmdusingknitr::rmarkdown

Last update: 2023-06-05
Started: 2020-04-13

Evaluate by ID/group
Introduction | Averaging | Voting | ID evaluation with evaluate() | Using voting | Per model | In 'gaussian' evaluation | Results for each ID
Rendered fromevaluate_by_id.Rmdusingknitr::rmarkdown

Last update: 2023-06-05
Started: 2020-04-13

Multiple-k: Picking the number of folds for cross-validation
Introduction | Does it matter? | Multiple-k repeated cross-validation | Creating folds | Performing cross-validation | Inspecting the effect of k | Class level results
Rendered frompicking_the_number_of_folds_for_cross-validation.Rmdusingknitr::rmarkdown

Last update: 2023-06-05
Started: 2021-03-07

Readme and manuals

Help Manual

Help pageTopics
cvms: A package for cross-validating regression and classification modelscvms-package cvms
Create baseline evaluationsbaseline
Create baseline evaluations for binary classificationbaseline_binomial
Create baseline evaluations for regression modelsbaseline_gaussian
Create baseline evaluationsbaseline_multinomial
Select metrics for binomial evaluationbinomial_metrics
Generate model formulas by combining predictorscombine_predictors generate_formulas
Compatible formula termscompatible.formula.terms
Create a confusion matrixconfusion_matrix
Cross-validate regression models for model selectioncross_validate
Cross-validate custom model functions for model selectioncross_validate_fn
Create a list of dynamic font color settings for plotsdynamic_font_color_settings
Evaluate your model's performanceevaluate
Evaluate residuals from a regression taskevaluate_residuals
Create a list of font settings for plotsfont
Select metrics for Gaussian evaluationgaussian_metrics
Examples of model_fn functionsmodel_functions
Find the data points that were hardest to predicthardest most_challenging
Generate a multiclass probability tibblemulticlass_probability_tibble
Select metrics for multinomial evaluationmultinomial_metrics
Musician groupsmusicians
Participant scoresparticipant.scores
Plot a confusion matrixplot_confusion_matrix
Density plot for a metricplot_metric_density
Precomputed formulasprecomputed.formulas
Examples of predict_fn functionspredict_functions
Predicted musician groupspredicted.musicians
Examples of preprocess_fn functionspreprocess_functions
A set of process information object constructorsas.character.process_info_binomial as.character.process_info_gaussian as.character.process_info_multinomial print.process_info_binomial print.process_info_gaussian print.process_info_multinomial process_info_binomial process_info_gaussian process_info_multinomial
Reconstruct model formulas from results tibblesreconstruct_formulas
Select model definition columnsselect_definitions
Select columns with evaluation metrics and model definitionsselect_metrics
Simplify formula with inline functionssimplify_formula
Create a list of settings for the sum tiles in plot_confusion_matrix()sum_tile_settings
Summarize metrics with common descriptorssummarize_metrics
Check and update hyperparametersupdate_hyperparameters
Validate regression models on a test setvalidate
Validate a custom model function on a test setvalidate_fn
Wine varietieswines