pROC: display and analyze ROC curves in R

More screenshots and examples →

Citation

If you use pROC in published research, please cite the following paper:

Xavier Robin, Natacha Turck, Alexandre Hainard, Natalia Tiberti, Frédérique Lisacek, Jean-Charles Sanchez and Markus Müller (2011). pROC: an open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics, 12, p. 77. DOI: 10.1186/1471-2105-12-77.

Description

An R package to visualize, smooth and compare receiver operating characteristic (ROC) curves with statistical tests and confidence intervals.

Version

News

Authors

Xavier Robin, Natacha Turck, Alexandre Hainard, Natalia Tiberti, Frédérique Lisacek, Jean-Charles Sanchez and Markus Müller

Contributors

Stefan Siegert, Matthias Doering, Zane Billings

Report a bug

GitHub issue tracker

License

GPLv3

Finding help

Frequently asked questions
Check the manual, or type ?pROC on the R prompt.
If you still have questions, ask a question on Stack Overflow tagged with proc-r-package.

Installation

There is no need to download the package. The installation can be done in one command directly from R:

install.packages("pROC")

The package must then be loaded with:

library(pROC)

To get help, enter the following in the R prompt:

?pROC

Update

You can update pROC by installing it again:

install.packages("pROC")

Or updates all your packages:

update.packages()

Version history

1.19.0.1 (2025-07-31)

Move fun.sesp definition to work around LudvigOlsen/cvms#44.

1.19.0 (2025-07-30)

release notes

ci.coords can now take the same input values as coords (issue #90)
ci.coords can be plotted
Added "lr_pos" and "lr_neg" to coords (issue #102)
coords with partial.auc now interpolates bounds when needed
Added ignore.partial.auc argument to coords
Deprecated transpose, as.list, as.matrix and drop in coords
Deprecated the algorithm argument to roc and fun.sesp value
Deprecated the progress and parallel argument for bootstrap operations.
Removed dependencies on doParallel and retired package plyr (thanks to Michael Chirico, pr #134, #135, #136, #137, #138, #139 and #140).

1.18.5 (2023-07-04)

release notes

Fixed formula input when given as variable and combined with with (issue #111)
Fixed formula containing variables with spaces (issue #120)
Fixed broken grouping when colour argument was given in ggroc (issue #121)

1.18.4 (2023-07-04)

Fix broken legacy.axes in ggroc

1.18.3 (2023-07-02)

Fix warning about aes_string in ggroc
Improvements in tests (thanks to Michael Chirico, pr #114, #115)
Fix upcoming CRAN warning about numeric version comparisons

1.18.2 (2023-05-13)

Fix CRAN submission NOTE (issue #112)

1.18.1 (2023-05-07)

Fix printing some curves built from a formula (issue #101)
Fix tests following upstream updates in density()
Improvements in documentation

1.18.0 (2021-09-02)

release notes

Add CI of the estimate for roc.test (DeLong, paired only for now) (code contributed by Zane Billings) (pr #95)
Fix documentation and alternative hypothesis for Venkatraman test (issue #92)

1.17.0.1 (2021-01-07)

release notes

Fix CRAN incoming checks as requested by CRAN

1.17.0 (2020-12-29)

Accept more values in input of coords (issue #67)
Accept kappa for the power.roc.test of two ROC curves (issue #82)
The input argument to coords for smooth.roc curves no longer has a default
The x argument to coords for smooth.roc can now be set to all (also the default)
Fix bootstrap roc.test and cov with smooth.roc curves
The ggroc function can now plot smooth.roc curves (issue #86)
Remove warnings with warnPartialMatchDollar option (issue #87)
Make tests depending on vdiffr conditional (issue #88)

1.16.2 (2020-03-19)

Skip test depending on Rcpp version

1.16.1 (2020-01-13)

release notes

Skip timing-dependent test failing occasionally on CRAN

1.16.0 (2020-01-12)

release notes

BACKWARD INCOMPATIBLE CHANGE: transpose argument to coords switched to FALSE by default (issue #54)
BACKWARD INCOMPATIBLE CHANGE: ci.coords return value is now of list type and easier to use
Fix one-sided DeLong test for curves with direction=">" (issue #64)
Fix an error in ci.coords due to expected NA values in some coords (like "precision") (issue #65)
Ordrered predictors are converted to numeric in a more robust way (issue #63)
Cleaned up power.roc.test code (issue #50)
Fix pairing with roc.formula and warn if na.action is not set to "na.pass" or "na.fail" (issue #68)
Fix ci.coords not working with smooth.roc curves

1.15.3 (2019-07-21)

release notes

Fix: previous version accidentally set transpose = FALSE, should only be for next major release

1.15.2 (2019-07-20)

Fix -Inf threshold in coords for curves with direction = ">" (issue #60)
Restore lazy loading of the data and fix an R CMD check warning "Variables with usage in documentation object aSAH not in code"

1.15.1 (2019-07-17)

Fix erroneous error in ci.coords with ret="threshold" (issue #57)
Fix vdiffr unit tests with ggplot2 3.2.0 (issue #53)
Keep list order in ggroc (issue #58)

1.15.0 (2019-06-01)

release notes

roc now prints messages when autodetecting levels and direction by default. Turn off with quiet = TRUE or set these values explicitly
Speedup with algorithm = 2 (issue #44) and in coords (issue #52)
New algorithm = 6 (used by default) uses algorithm = 2 for numeric data, and algorithm = 3 for ordered vectors
New roc.data.frame method and roc_ function for use in pipelines
coords can now returns youden and closest.topleft values (issue #48)
New transpose argument for coords, TRUE by default (issue #54)
Use text instead of Tcl/Tk progress bar by default (issue #51)
Fix method = "density" smoothing when called directly from roc (issue #49)
Renamed roc argument n to smooth.n
Fixed are.paired ignoring smoothing arguments of roc2 with return.paired.rocs
New ret option "all" in coords (issue #47)
drop in coords now drops the dimension of ret too (#issue 43)

1.14.0 (2019-03-12)

release notes

The multiclass.roc function now accepts multivariate decision values (code contributed by Matthias Döring)
ggroc supports multiple aesthetics (issue #42)
Make ggplot2 dependency optional
CSuggested packages can be installed interactively when required
Passing both cases and controls or response and predictor arguments is now an error
Many small bug fixes

1.13.0 (2018-09-23)

release notes

roc now returns NaN when predictor contains infinite values (issue #30)
Better handling of near-ties near +-Infinity and 0
ggroc supports aes="group" to allow curves with identical aesthetics

1.12.1 (2018-05-06)

release notes

Fix a regression causing the allocation of a very large vector (issue #29)

1.12.0 (2018-05-05)

R release notes

Fix bug that crashed DeLong calculations when predictor had near-ties close to the floating point precision limit that were rounded back to a predictor value (issue #25)
Fix bug that crashed ci.auc and var if direction was ">" and percent=TRUE (issue #25)
Fix bug causing ci to return NaN values with method="delong" when cases or controls had a single observation (issue #27)
Fix power.roc.curve failed with curves having percent=TRUE
Fix ci(..., of="coords") returned the ci function instead of the CI
C++ code now check for user interrupts regularly with Rcpp::checkUserInterrupt()
Better error message for ci.coords attempting to return threshold
New algorithm = 5 (used by default) chooses the algorithm based on the number of thresholds to avoid worst case with algorithm = 3

1.11.0 (2018-03-24)

R release notes

Added argument legacy.axes to ggroc
Fix NOTE about apparent S3 methods exported but not registered in R CMD check

1.10.0 (2017-06-10)

R release notes

Basic ggplot2 support (one and multiple ROC curves)
Implement precision and recall for coords
Fix: properly handle NAs in cases when passing cases/controls to roc (thanks Thomas König for the report)
Fix various minor bugs detected with new unit tests

1.9.1 (2017-02-05)

R release notes

Fix: subset and na.action arguments now handled properly in roc.formula (thanks Terry Therneau for the report)
Added policies to handle the case where a ROC curve has multiple "best" threshold in ci (thanks Nicola Toschi for the report)
Support xlim and ylim gracefully in plot.roc
Improved validation of input class levels and direction; messages when auto-detecting, use quiet to turn on
Removed extraneous name attribute on the p.value (thanks Paweł Kleka for the report)
Faster DeLong algorithm (code contributed by Stefan Siegert)

1.8 (2015-05-04)

NAMESPACE now properly exports the methods as S3 methods.
Now uses and works with requireNamespace
Add ability to supply two ordered factors with identical levels as control / cases
Deprecate dangerous use of a matrix as response / predictor with a warning
Forward best.method and best.weights arguments to coords in ci.thresholds
Fix CITATION file as per CRAN request
pAUC correction is undefined for partial AUC below the diagonal (result < 0.5) and now returns NA (with a warning). Thanks Vincenzo Lagani for the report.

1.7.3 (2014-06-14)

R release notes

Fixed AUC of binomial-smoothed ROC off by 100^2 (thanks Bao-Li Chang for the report)
Fix print of logcondens-smoothed ROC

1.7.2 (2014-04-05)

R release notes

Fixed bug where ci.coords with x="best" would fail if one or more resampled ROC curve had multiple "best" thresholds (thanks Berend Terluin for the report)
Fixed bug in ci.coords: passing more than one value in x now works
Fixed typo in documentation of direction argument to roc (thanks Le Kang for the report)
Add a warning when computing statistics of ROC curve with AUC = 1
Require latest version of Rcpp to avoid weird errors (thanks Tom Liptrot for the report)

1.7.1 (2014-02-20)

release notes

Close SOCK cluster on Windows with parallel=TRUE
Fixed really use algorithm 1 when microbenchmark fails

1.7 (2014-02-19)

release notes

Faster algorithm for DeLong roc.test, power.roc.test, ci.auc, var and cov function (no large matrix allocation)
Handling Math and Operations correctly on auc and ci objects (see ?groupGeneric.pROC)
The formula for roc.formula can now provide several predictors and a list of ROC curves will be returned
Fixed documentation of ci.coords with examples
Fixed binormal AUC computed with triangulation despite the claim in the documentation
Fixed unstated requirement on Rcpp >= 0.10.5

1.6.0.1 (2013-12-28)

release notes

Removed erroneous error message displayed when predictors and responses were not vectors.

1.6 (2013-12-26)

release notes

New power.roc.test function for sample size and power computations.
New cov and var functions supports new "obuchowski" method.
New ci.coords function to compute CI of arbitrary coords.
coords accepts new ret value "1-accuracy".
Introducing various algorithms to compute sensitivities and specificites, with a more vectorized code or Rcpp. See algorithm in ?roc for more details on the trade-offs of the different methods.
Faster algorithm for DeLong roc.test, ci, var and cov function (thanks Kazuki Yoshida).
are.paired now also checks for identical levels.
Fixed a warning generated in the examples.
Fixed several bugs related with smooth.roc curves.
Additional input data sanity checks.
Now requires R ≥ 2.13 (in fact, since 1.5.1, thanks Emmanuel Curis for the report).
Progress bars now defaults to text on Macs where tcltk seems broken (thanks Gerard Smits for the report).

1.5.4 (2012-08-31)

Running less smooth.roc examples with logcondens because they take too much time (requested by Uwe Ligges).

1.5.3 (2011-08-31)

release notes

AUC specification was lost when roc.test, cov or var was passed an auc object.
Correct computation of "accuracy" in coords (thanks to Kosuke Yoshihara for the report).

1.5.1 (2011-03-09)

release notes

Faster loading of the package (thanks to Prof Brian Ripley and Glenn Lawyer for the report).

1.5 (2011-12-12)

release notes

New cov and var functions.
coords accepts new ret values: "accuracy", "tn", "tp", "fn", "fp", "npv", "ppv", "1-specificity", "1-sensitivity", "1-npv", "1-ppv", "npe" and "ppe".
New legacy.axes argument to plot 1-specificity rather than specificity.
New axes argument to turn off the plotting of the axis.
New logcondens and logcondens.smooth (Univariate Log-Concave Density Estimation) smoothing methods.
New function has.partial.auc to determine if an AUC is full or partial.
New argument drop for coords.
auc and multiclass.auc objects now also have secondary class numeric.
Updated load call.
Delong's CI reversed in ROC curves with direction=">".
Delong's CI AUC returned values > 1 or < 0 in some rare cases.
Minor improvements in documentation.

1.4.9 (2014-01-04)

S+

release notes

This is the last version of pROC for S+. It introduces some features of pROC 1.5 (covariance and variance functions, some bugfixes) and 1.6 (power tests) for R into S+, and will not be maintained any more.
Fixed custom output report.
New power.roc.test function for sample size and power computations.
New cov and var functions.
auc objects now also have secondary class numeric.
Delong's CI reversed in ROC curves with direction=">"
Delong's CI AUC returned values > 1 or < 0 in some rare cases
Delong's CI AUC returned NA values when AUC was 1. Now reporting 1-1 interval.
Minor improvements in documentation.

1.4.4 (2011-08-10)

S+

release notes

Fixed alternative for one-tailed tests.
Removed COPYING file to fix a warning in r-devel.

1.4.3 (2011-03-18)

S+

release notes

Updated citation.

1.4.2 (2011-03-03)

S+

release notes

Fixed bootstrap roc.test generating NAs when smooth.rocs were used with reuse.auc=FALSE (thanks to Buddy for the report).
Documented a warning that was missing in roc.test.
Updated citation.

1.4.1 (2011-01-27)

S+

release notes

Venkatraman's test for unpaired ROC curves.

1.4 (2011-01-21)

S+

release notes

'smooth' does not apply on ordered factors anymore.
Multi-class AUC support (R only).
Can choose how best thresold is determined (best.method and best.weights in coords and print.thres.best.method and print.thres.best.weights in plot.roc).
Minor fixes in documentation (R and S+) and citation (S+ only).
print now prints the response instead of response and more informative data in htests.
Bootstrap with ci.auc consumes much less memory.
Unpaired bootstrap and DeLong's test.
Specificity and sensitivity tests (in roc.test).

1.3.2 (2010-08-24)

S+

release notes

print.auc printed incorrect CI in plot.roc (thanks to Alexander B. Leichtle for the report).
Failed to detect local maximas in coords when 2 or less points were selected.
Don't consider ROC extremities (+-Inf at 1.0/0.0 SE<->SP) as local maximas.

1.3.1 (2010-08-18)

release notes

Sensitivity and specificity were inverted in coords when results were reported as list.
Faster checks with \dontrun{} in roc.test.

1.3 (2010-08-13)

S+

release notes

CI is not re-computed by default in smooth.roc. You can still turn it on with reuse.ci=TRUE.
New function are.paired.
Local maximas could be incorrectly detected in coords (and plot.roc) with predictors containing more than 2 levels.
New method venkatraman for roc.test.
MASS and tcltk packages are now only suggested instead of required (R only).
... not passed correctly in plot.ci.se with type="bars" resulting in an error (R only).

1.2.1 (2010-05-11)

S+

release notes

Handle method arguments for smooth.roc and ci.auc separately in roc.default (R-only).
Added auc.polygon.* and max.auc.polygon.* arguments for polygon in plot.roc.

1.2 (2010-05-09)

release notes

Added DeLong method in ci.auc (with a GUI, S+ only).
Return value of ci.auc does not contain an "aucs" item anymore.
Put most examples with bootstrap within \dontrun{} blocks for faster (but less useful) checks execution.

1.1 (2010-05-05)

S+

release notes

Added lines.roc functions for ROC.
Added type argument for both lines.roc and plot.roc.
Added print.auc.col argument to plot.roc.
Fixed a warning in roc.test.default when the class of predictor1 had several elements.
Fixed an encoding failure during the checks on MacOS X (R only).

1.0.1 (2010-04-28)

release notes

Reduced examples execution time. Added low boot.n in the slowest examples and reuse.auc and reuse.ci arguments in smooth.roc.roc.

1.0 (2010-04-27)

S+

release notes

First public release.

References

James Carpenter and John Bithell (2000) "Bootstrap condence intervals: when, which, what? A practical guide for medical statisticians". Statistics in Medicine 19, 1141–1164. DOI: 10.1002/(SICI)1097-0258(20000515)19:9<1141::AID-SIM479>3.0.CO;2-F; PMID: 10797513.
Elisabeth R. DeLong, David M. DeLong and Daniel L. Clarke-Pearson (1988) "Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach". Biometrics 44, 837–845. JSTOR: 2531595; PMID: 3203132.
Lutz Duembgen, Kaspar Rufibach (2011) "logcondens: Computations Related to Univariate Log-Concave Density Estimation". Journal of Statistical Software, 39, 1–28. URL: jstatsoft.org/v39/i06.
Tom Fawcett (2006) "An introduction to ROC analysis". Pattern Recognition Letters 27, 861–874. DOI: j.patrec.2005.10.010.
James E. Hanley (1988) "The robustness of the "binormal" assumptions used in fitting ROC curves". Medical Decision Making 8, 197–203. DOI: 10.1177/0272989X8800800308; PMID: 3398748.
Donna Katzman McClish (1989) "Analyzing a Portion of the ROC Curve". Medical Decision Making 9, 190–195. DOI: 10.1177/0272989X8900900307; PMID: 2668680.
Nancy A. Obuchowski and Donna K. McClish (1997) "Sample size determination for diagnostic accurary studies involving binormal ROC curve indices". Statistics in Medicine, 16, 1529–1542. DOI: 10.1002/(SICI)1097-0258(19970715)16:13<1529::AID-SIM565>3.0.CO;2-H.
Nancy A. Obuchowski, Micharl L. Lieber and Frank H. Wians Jr. (2004) "ROC Curves in Clinical Chemistry: Uses, Misuses, and Possible Solutions". Clinical Chemistry, 50, 1118–1125. DOI: 10.1373/clinchem.2004.031823.
Margaret Pepe, Gary Longton and Holly Janes (2009) "Estimation and Comparison of Receiver Operating Characteristic Curves". The Stata journal 9, 1. PMID: 20161343.
Xavier Robin, Natacha Turck et al. (2011) "pROC: an open-source package for R and S+ to analyze and compare ROC curves". BMC Bioinformatics, 12, 77. DOI: 10.1186/1471-2105-12-77.
E. S. Venkatraman and Colin B. Begg (1996) "A distribution-free procedure for comparing receiver operating characteristic curves from a paired experiment". Biometrika 83, 835–848. DOI: 10.1093/biomet/83.4.835
E. S. Venkatraman (2000) "A Permutation Test to Compare Receiver Operating Characteristic Curves". Biometrics 56, 1134–1138. DOI: 10.1111/j.0006-341X.2000.01134.x.
Kelly H. Zou, W. J. Hall and David E. Shapiro (1997) "Smooth non-parametric receiver operating characteristic (ROC) curves for continuous diagnostic tests". Statistics in Medicine 18, 2143–2156. DOI: 10.1002/(SICI)1097-0258(19971015)16:19<2143::AID-SIM655>3.0.CO;2-3; PMID: 9330425 .

S+

The pROC package initially supported S+ as well as R. However, TIBCO S+ no longer supports packages and discontinued the CSAN. As a result pROC is no longer supported for S+. The S+ installation page is provided for historical purpose only.