Citation
If you use pROC in published research, please cite the following paper:
Xavier Robin, Natacha Turck, Alexandre Hainard, Natalia Tiberti, Frédérique Lisacek, Jean-Charles Sanchez and Markus Müller (2011). pROC: an open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics, 12, p. 77. DOI: 10.1186/1471-2105-12-77.
Description
An R package to visualize, smooth and compare receiver operating characteristic (ROC) curves with statistical tests and confidence intervals.
Authors
Xavier Robin, Natacha Turck, Alexandre Hainard, Natalia Tiberti, Frédérique Lisacek, Jean-Charles Sanchez and Markus Müller
Contributors
Stefan Siegert, Matthias Doering, Zane Billings
Report a bug
License
Finding help
- Frequently asked questions
- Check the manual, or type
?pROCon the R prompt. - If you still have questions, ask a question on Stack Overflow tagged with
proc-r-package.
Installation
There is no need to download the package. The installation can be done in one command directly from R:
install.packages("pROC")
The package must then be loaded with:
library(pROC)
To get help, enter the following in the R prompt:
?pROC
Update
You can update pROC by installing it again:
install.packages("pROC")
Or updates all your packages:
update.packages()
Version history
1.19.0.1 (2025-07-31)
- Move
fun.sespdefinition to work around LudvigOlsen/cvms#44.
1.19.0 (2025-07-30)
ci.coordscan now take the sameinputvalues ascoords(issue #90)ci.coordscan beplotted- Added "lr_pos" and "lr_neg" to
coords(issue #102) coordswith partial.auc now interpolates bounds when needed- Added
ignore.partial.aucargument tocoords - Deprecated
transpose,as.list,as.matrixanddropincoords - Deprecated the
algorithmargument torocandfun.sespvalue - Deprecated the
progressandparallelargument for bootstrap operations. - Removed dependencies on doParallel and retired package plyr (thanks to Michael Chirico, pr #134, #135, #136, #137, #138, #139 and #140).
1.18.5 (2023-07-04)
- Fixed formula input when given as variable and combined with
with(issue #111) - Fixed formula containing variables with spaces (issue #120)
- Fixed broken grouping when
colourargument was given inggroc(issue #121)
1.18.4 (2023-07-04)
- Fix broken
legacy.axesinggroc
1.18.3 (2023-07-02)
1.18.2 (2023-05-13)
- Fix CRAN submission NOTE (issue #112)
1.18.1 (2023-05-07)
- Fix
printing some curves built from a formula (issue #101) - Fix tests following upstream updates in density()
- Improvements in documentation
1.18.0 (2021-09-02)
1.17.0.1 (2021-01-07)
- Fix CRAN incoming checks as requested by CRAN
1.17.0 (2020-12-29)
- Accept more values in
inputof coords (issue #67) - Accept
kappafor thepower.roc.testof two ROC curves (issue #82) - The
inputargument tocoordsforsmooth.roccurves no longer has a default - The
xargument tocoordsforsmooth.roccan now be set toall(also the default) - Fix bootstrap roc.test and cov with
smooth.roccurves - The
ggrocfunction can now plotsmooth.roccurves (issue #86) - Remove warnings with
warnPartialMatchDollaroption (issue #87) - Make tests depending on vdiffr conditional (issue #88)
1.16.2 (2020-03-19)
- Skip test depending on Rcpp version
1.16.1 (2020-01-13)
- Skip timing-dependent test failing occasionally on CRAN
1.16.0 (2020-01-12)
- BACKWARD INCOMPATIBLE CHANGE:
transposeargument tocoordsswitched to FALSE by default (issue #54) - BACKWARD INCOMPATIBLE CHANGE:
ci.coordsreturn value is now of list type and easier to use - Fix one-sided DeLong test for curves with
direction=">"(issue #64) - Fix an error in
ci.coordsdue to expected NA values in some coords (like "precision") (issue #65) - Ordrered predictors are converted to numeric in a more robust way (issue #63)
- Cleaned up
power.roc.testcode (issue #50) - Fix pairing with
roc.formulaand warn ifna.actionis not set to "na.pass" or "na.fail" (issue #68) - Fix
ci.coordsnot working withsmooth.roccurves
1.15.3 (2019-07-21)
- Fix: previous version accidentally set transpose = FALSE, should only be for next major release
1.15.2 (2019-07-20)
- Fix -Inf threshold in coords for curves with
direction = ">"(issue #60) - Restore lazy loading of the data and fix an
R CMD checkwarning "Variables with usage in documentation objectaSAHnot in code"
1.15.1 (2019-07-17)
1.15.0 (2019-06-01)
rocnow prints messages when autodetectinglevelsanddirectionby default. Turn off withquiet = TRUEor set these values explicitly- Speedup with
algorithm = 2(issue #44) and incoords(issue #52) - New
algorithm = 6(used by default) usesalgorithm = 2for numeric data, andalgorithm = 3for ordered vectors - New
roc.data.framemethod androc_function for use in pipelines coordscan now returnsyoudenandclosest.topleftvalues (issue #48)- New
transposeargument forcoords, TRUE by default (issue #54) - Use text instead of Tcl/Tk progress bar by default (issue #51)
- Fix
method = "density"smoothing when called directly fromroc(issue #49) - Renamed
rocargumentntosmooth.n - Fixed
are.pairedignoring smoothing arguments ofroc2withreturn.paired.rocs - New
retoption "all" incoords(issue #47) dropincoordsnow drops the dimension ofrettoo (#issue 43)
1.14.0 (2019-03-12)
- The
multiclass.rocfunction now accepts multivariate decision values (code contributed by Matthias Döring) ggrocsupports multiple aesthetics (issue #42)- Make
ggplot2dependency optional - CSuggested packages can be installed interactively when required
- Passing both
casesandcontrolsorresponseandpredictorarguments is now an error - Many small bug fixes
1.13.0 (2018-09-23)
rocnow returnsNaNwhen predictor contains infinite values (issue #30)- Better handling of near-ties near +-Infinity and 0
ggrocsupports aes="group" to allow curves with identical aesthetics
1.12.1 (2018-05-06)
- Fix a regression causing the allocation of a very large vector (issue #29)
1.12.0 (2018-05-05)
- Fix bug that crashed DeLong calculations when predictor had near-ties close to the floating point precision limit that were rounded back to a predictor value (issue #25)
- Fix bug that crashed
ci.aucandvarifdirectionwas ">" andpercent=TRUE(issue #25) - Fix bug causing
cito returnNaNvalues withmethod="delong"when cases or controls had a single observation (issue #27) - Fix
power.roc.curvefailed with curves havingpercent=TRUE - Fix
ci(..., of="coords")returned thecifunction instead of the CI - C++ code now check for user interrupts regularly with Rcpp::checkUserInterrupt()
- Better error message for
ci.coordsattempting to returnthreshold - New algorithm = 5 (used by default) chooses the algorithm based on the number of thresholds to avoid worst case with algorithm = 3
1.11.0 (2018-03-24)
- Added argument
legacy.axestoggroc - Fix NOTE about
apparent S3 methods exported but not registeredinR CMD check
1.10.0 (2017-06-10)
- Basic ggplot2 support (one and multiple ROC curves)
- Implement
precisionandrecallforcoords - Fix: properly handle NAs in cases when passing cases/controls to
roc(thanks Thomas König for the report) - Fix various minor bugs detected with new unit tests
1.9.1 (2017-02-05)
- Fix:
subsetandna.actionarguments now handled properly inroc.formula(thanks Terry Therneau for the report) - Added policies to handle the case where a ROC curve has multiple "best" threshold in
ci(thanks Nicola Toschi for the report) - Support
xlimandylimgracefully inplot.roc - Improved validation of input class
levelsanddirection; messages when auto-detecting, usequietto turn on - Removed extraneous
nameattribute on thep.value(thanks Paweł Kleka for the report) - Faster DeLong algorithm (code contributed by Stefan Siegert)
1.8 (2015-05-04)
- NAMESPACE now properly exports the methods as S3 methods.
- Now uses and works with
requireNamespace - Add ability to supply two ordered factors with identical levels as control / cases
- Deprecate dangerous use of a matrix as response / predictor with a warning
- Forward
best.methodandbest.weightsarguments tocoordsinci.thresholds - Fix CITATION file as per CRAN request
- pAUC correction is undefined for partial AUC below the diagonal (result < 0.5) and now returns NA (with a warning). Thanks Vincenzo Lagani for the report.
1.7.3 (2014-06-14)
- Fixed AUC of binomial-smoothed ROC off by 100^2 (thanks Bao-Li Chang for the report)
- Fix print of logcondens-smoothed ROC
1.7.2 (2014-04-05)
- Fixed bug where
ci.coordswithx="best"would fail if one or more resampled ROC curve had multiple "best" thresholds (thanks Berend Terluin for the report) - Fixed bug in
ci.coords: passing more than one value inxnow works - Fixed typo in documentation of
directionargument toroc(thanks Le Kang for the report) - Add a warning when computing statistics of ROC curve with AUC = 1
- Require latest version of Rcpp to avoid weird errors (thanks Tom Liptrot for the report)
1.7.1 (2014-02-20)
- Close SOCK cluster on Windows with parallel=TRUE
- Fixed really use algorithm 1 when microbenchmark fails
1.7 (2014-02-19)
- Faster algorithm for DeLong
roc.test,power.roc.test,ci.auc,varandcovfunction (no large matrix allocation) - Handling Math and Operations correctly on
aucandciobjects (see?groupGeneric.pROC) - The
formulaforroc.formulacan now provide several predictors and a list of ROC curves will be returned - Fixed documentation of
ci.coordswith examples - Fixed binormal AUC computed with triangulation despite the claim in the documentation
- Fixed unstated requirement on Rcpp >= 0.10.5
1.6.0.1 (2013-12-28)
- Removed erroneous error message displayed when predictors and responses were not vectors.
1.6 (2013-12-26)
- New
power.roc.testfunction for sample size and power computations. - New
covandvarfunctions supports new"obuchowski"method. - New
ci.coordsfunction to compute CI of arbitrary coords. coordsaccepts newretvalue "1-accuracy".- Introducing various algorithms to compute sensitivities and specificites, with a more vectorized code or Rcpp. See
algorithmin?rocfor more details on the trade-offs of the different methods. - Faster algorithm for DeLong
roc.test,ci,varandcovfunction (thanks Kazuki Yoshida). are.pairednow also checks for identicallevels.- Fixed a warning generated in the examples.
- Fixed several bugs related with
smooth.roccurves. - Additional input data sanity checks.
- Now requires R ≥ 2.13 (in fact, since 1.5.1, thanks Emmanuel Curis for the report).
- Progress bars now defaults to text on Macs where
tcltkseems broken (thanks Gerard Smits for the report).
1.5.4 (2012-08-31)
- Running less smooth.roc examples with logcondens because they take too much time (requested by Uwe Ligges).
1.5.3 (2011-08-31)
- AUC specification was lost when
roc.test,covorvarwas passed anaucobject. - Correct computation of "accuracy" in
coords(thanks to Kosuke Yoshihara for the report).
1.5.1 (2011-03-09)
- Faster loading of the package (thanks to Prof Brian Ripley and Glenn Lawyer for the report).
1.5 (2011-12-12)
- New
covandvarfunctions. coordsaccepts newretvalues: "accuracy", "tn", "tp", "fn", "fp", "npv", "ppv", "1-specificity", "1-sensitivity", "1-npv", "1-ppv", "npe" and "ppe".- New
legacy.axesargument toplot1-specificity rather than specificity. - New
axesargument to turn off the plotting of the axis. - New
logcondensandlogcondens.smooth(Univariate Log-Concave Density Estimation) smoothing methods. - New function
has.partial.aucto determine if an AUC is full or partial. - New argument
dropforcoords. aucandmulticlass.aucobjects now also have secondary classnumeric.- Updated load call.
- Delong's CI reversed in ROC curves with
direction=">". - Delong's CI AUC returned values > 1 or < 0 in some rare cases.
- Minor improvements in documentation.
1.4.9 (2014-01-04)
- This is the last version of pROC for S+. It introduces some features of pROC 1.5 (covariance and variance functions, some bugfixes) and 1.6 (power tests) for R into S+, and will not be maintained any more.
- Fixed custom output report.
- New
power.roc.testfunction for sample size and power computations. - New
covandvarfunctions. aucobjects now also have secondary classnumeric.- Delong's CI reversed in ROC curves with
direction=">" - Delong's CI AUC returned values > 1 or < 0 in some rare cases
- Delong's CI AUC returned NA values when AUC was 1. Now reporting 1-1 interval.
- Minor improvements in documentation.
1.4.4 (2011-08-10)
- Fixed alternative for one-tailed tests.
- Removed COPYING file to fix a warning in r-devel.
1.4.3 (2011-03-18)
- Updated citation.
1.4.2 (2011-03-03)
- Fixed bootstrap
roc.testgenerating NAs whensmooth.rocs were used withreuse.auc=FALSE(thanks to Buddy for the report). - Documented a warning that was missing in
roc.test. - Updated citation.
1.4.1 (2011-01-27)
- Venkatraman's test for unpaired ROC curves.
1.4 (2011-01-21)
- 'smooth' does not apply on
orderedfactors anymore. - Multi-class AUC support (R only).
- Can choose how
bestthresold is determined (best.methodandbest.weightsincoordsandprint.thres.best.methodandprint.thres.best.weightsinplot.roc). - Minor fixes in documentation (R and S+) and
citation(S+ only). printnow prints the response instead of response and more informative data inhtests.- Bootstrap with
ci.aucconsumes much less memory. - Unpaired bootstrap and DeLong's test.
- Specificity and sensitivity tests (in
roc.test).
1.3.2 (2010-08-24)
print.aucprinted incorrect CI inplot.roc(thanks to Alexander B. Leichtle for the report).- Failed to detect local maximas in
coordswhen 2 or less points were selected. - Don't consider ROC extremities (+-Inf at 1.0/0.0 SE<->SP) as local maximas.
1.3.1 (2010-08-18)
- Sensitivity and specificity were inverted in coords when results were reported as list.
- Faster checks with
\dontrun{}inroc.test.
1.3 (2010-08-13)
- CI is not re-computed by default in
smooth.roc. You can still turn it on withreuse.ci=TRUE. - New function
are.paired. - Local maximas could be incorrectly detected in
coords(andplot.roc) withpredictors containing more than 2 levels. - New method
venkatramanforroc.test. - MASS and tcltk packages are now only suggested instead of required (R only).
...not passed correctly inplot.ci.sewithtype="bars"resulting in an error (R only).
1.2.1 (2010-05-11)
- Handle
methodarguments forsmooth.rocandci.aucseparately inroc.default(R-only). - Added
auc.polygon.*andmax.auc.polygon.*arguments forpolygoninplot.roc.
1.2 (2010-05-09)
- Added DeLong method in
ci.auc(with a GUI, S+ only). - Return value of ci.auc does not contain an
"aucs"item anymore. - Put most examples with bootstrap within
\dontrun{}blocks for faster (but less useful) checks execution.
1.1 (2010-05-05)
- Added
lines.rocfunctions for ROC. - Added
typeargument for bothlines.rocandplot.roc. - Added
print.auc.colargument toplot.roc. - Fixed a warning in
roc.test.defaultwhen the class ofpredictor1had several elements. - Fixed an encoding failure during the checks on MacOS X (R only).
1.0.1 (2010-04-28)
- Reduced examples execution time. Added low
boot.nin the slowest examples andreuse.aucandreuse.ciarguments insmooth.roc.roc.
1.0 (2010-04-27)
- First public release.
References
- James Carpenter and John Bithell (2000) "Bootstrap condence intervals: when, which, what? A practical guide for medical statisticians". Statistics in Medicine 19, 1141–1164. DOI: 10.1002/(SICI)1097-0258(20000515)19:9<1141::AID-SIM479>3.0.CO;2-F; PMID: 10797513.
- Elisabeth R. DeLong, David M. DeLong and Daniel L. Clarke-Pearson (1988) "Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach". Biometrics 44, 837–845. JSTOR: 2531595; PMID: 3203132.
- Lutz Duembgen, Kaspar Rufibach (2011) "logcondens: Computations Related to Univariate Log-Concave Density Estimation". Journal of Statistical Software, 39, 1–28. URL: jstatsoft.org/v39/i06.
- Tom Fawcett (2006) "An introduction to ROC analysis". Pattern Recognition Letters 27, 861–874. DOI: j.patrec.2005.10.010.
- James E. Hanley (1988) "The robustness of the "binormal" assumptions used in fitting ROC curves". Medical Decision Making 8, 197–203. DOI: 10.1177/0272989X8800800308; PMID: 3398748.
- Donna Katzman McClish (1989) "Analyzing a Portion of the ROC Curve". Medical Decision Making 9, 190–195. DOI: 10.1177/0272989X8900900307; PMID: 2668680.
- Nancy A. Obuchowski and Donna K. McClish (1997) "Sample size determination for diagnostic accurary studies involving binormal ROC curve indices". Statistics in Medicine, 16, 1529–1542. DOI: 10.1002/(SICI)1097-0258(19970715)16:13<1529::AID-SIM565>3.0.CO;2-H.
- Nancy A. Obuchowski, Micharl L. Lieber and Frank H. Wians Jr. (2004) "ROC Curves in Clinical Chemistry: Uses, Misuses, and Possible Solutions". Clinical Chemistry, 50, 1118–1125. DOI: 10.1373/clinchem.2004.031823.
- Margaret Pepe, Gary Longton and Holly Janes (2009) "Estimation and Comparison of Receiver Operating Characteristic Curves". The Stata journal 9, 1. PMID: 20161343.
- Xavier Robin, Natacha Turck et al. (2011) "pROC: an open-source package for R and S+ to analyze and compare ROC curves". BMC Bioinformatics, 12, 77. DOI: 10.1186/1471-2105-12-77.
- E. S. Venkatraman and Colin B. Begg (1996) "A distribution-free procedure for comparing receiver operating characteristic curves from a paired experiment". Biometrika 83, 835–848. DOI: 10.1093/biomet/83.4.835
- E. S. Venkatraman (2000) "A Permutation Test to Compare Receiver Operating Characteristic Curves". Biometrics 56, 1134–1138. DOI: 10.1111/j.0006-341X.2000.01134.x.
- Kelly H. Zou, W. J. Hall and David E. Shapiro (1997) "Smooth non-parametric receiver operating characteristic (ROC) curves for continuous diagnostic tests". Statistics in Medicine 18, 2143–2156. DOI: 10.1002/(SICI)1097-0258(19971015)16:19<2143::AID-SIM655>3.0.CO;2-3; PMID: 9330425 .
S+
The pROC package initially supported S+ as well as R. However, TIBCO S+ no longer supports packages and discontinued the CSAN. As a result pROC is no longer supported for S+. The S+ installation page is provided for historical purpose only.