The goal of parTimeROC is to store methods and procedures needed to run the time-dependent ROC analysis parametrically. This package adopts two different theoretical framework to produce the ROC curve which are from the proportional hazard model and copula function. Currently, this package only able to run analysis for single covariate/biomarker with survival time. The future direction for this work is to be able to include analysis for multiple biomarkers with longitudinal measurements.
You can install the development version of parTimeROC from GitHub with:
# The easiest way to get parTimeROC is to install:
install.packages("parTimeROC")
# Alternatively, install the development version from GitHub:
devtools::install_github("FaizAzhar/parTimeROC")Since this package also include the bayesian estimation procedure (rstan), please ensure to follow the correct installation setup such as demonstrated in this article.
A receiver operating characteristics (ROC) curve is a curve that measures a model’s accuracy to correctly classify a population into a binary status (eg: dead/alive). The curve acts as a tool for analysts to compare which model is suitable to be used as a classifiers. However, in survival analysis, it is noted that the status of population fluctuate across time. Thus, a standard ROC analysis might underestimates the true accuracy measurement that the classification model have. In a situation where the population might enter or exit any of the two status over time, including the time component into the ROC analysis is shown to be superior and can help analysts to assess the performance of the model’s accuracy over time. In addition, a time-dependent ROC can also show at which specific time point a model will have a similar performance measurement with other model.
For the time being, two methods are frequently used when producing the time-dependent ROC curve. The first method employs the Cox proportional hazard model (PH) to estimate the joint distribution of the covariates and time-to-event. The second method employs a copula function which link the marginal distributions of covariates and time-to-event to estimate its joint distribution. After obtaining estimates for the joint distribution, two metrics can be computed which is the time-dependent sensitivity and specificity. Plotting these two informations will generate the desired time-dependent ROC curve.
Explanations below are showing the functions that can be found within
parTimeROC package and its implementation.
timeroc_objFollowing an OOP approaches, a TimeROC object can be
initialized by using the parTimeROC::timeroc_obj()
method.
test <- parTimeROC::timeroc_obj("normal-gompertz-PH")
print(test)
#> Model Assumptions: Proportional Hazard (PH)
#> X : Gaussian
#> Time-to-Event : Gompertz
test <- parTimeROC::timeroc_obj("normal-gompertz-copula", copula = "gumbel90")
print(test)
#> Model Assumptions: 90 Degrees Rotated Gumbel Copula
#> X : Gaussian
#> Time-to-Event : GompertzNotice that we included the print method to generate the summary for
the test object which has a TimeROC class.
A list of distributions and copula have been stored within this
package. It is accessible via the get.distributions or
get.copula script.
names(parTimeROC::get.distributions)
#> [1] "exponential" "weibull" "gaussian" "normal" "lognormal"
#> [6] "gompertz" "skewnormal" "pch" "llogis"
names(parTimeROC::get.copula)
#> [1] "gaussian" "clayton90" "gumbel90" "gumbel" "joe90"rtimerocCommon tasks in mathematical modelling are prepared. For simulation
purposes, procedure to generate random data from PH or copula function
is created. The random data can be obtained using the
parTimeROC::rtimeroc(). The
parTimeROC::rtimeroc() returns a dataframe of 3 columns (t,
x, status).
library(parTimeROC)
## PH model
test <- timeroc_obj(dist = 'weibull-gompertz-PH')
set.seed(23456)
rr <- rtimeroc(obj = test, censor.rate = 0.5, n=500,
params.t = c(shape=2, rate=1),
params.x = c(shape=2, scale=1),
params.ph=0.5)
plot(t~x, rr)
Fig.1. Random data of biomarker and time-to-event
timeroc_fitWe can also fit datasets that have time-to-event, covariates and
status columns with the PH or copula model using the
parTimeROC::timeroc_fit().
For PH model, two fitting processes are done. One is to fit the biomarker distribution alone. Another is to fit the time-to-event that is assumed to follow a proportional hazard model.
Meanwhile, for copula method, the IFM technique is used due to its light computational requirement. Three fitting processes are conducted. One is to fit the marginal distribution for biomarker, another is to fit the marginal time-to-event. And lastly is to fit the copula function.
User can choose to conduct the model fitting procedure based on the
frequentist or bayesian approach by specifying the
method = 'mle' or method = 'bayes' within the
parTimeROC::timeroc_fit() function.
By default, the frequentist approach is used to estimate the model’s parameters.
library(parTimeROC)
## fitting copula model
test <- timeroc_obj(dist = 'gompertz-gompertz-copula', copula = "gumbel90")
set.seed(23456)
rr <- rtimeroc(obj = test, censor.rate = 0, n=500,
params.t = c(shape=3,rate=1),
params.x = c(shape=1,rate=2),
params.copula=-5) # name of parameter must follow standard
cc <- timeroc_fit(rr$x, rr$t, rr$event, obj = test)
print(cc)
#> Model: gompertz-gompertz-copula
#> ------
#> X (95% CI) :
#> AIC = -65.51402
#> est low upper se
#> shape 0.9342836 0.6216374 1.246930 0.1595163
#> rate 2.0930839 1.7975025 2.388665 0.1508096
#> ------
#> Time-to-Event (95% CI) :
#> AIC = -141.7148
#> est low upper se
#> shape 3.0894231 2.7066350 3.472211 0.19530364
#> rate 0.9160331 0.7546502 1.077416 0.08233974
#> ------
#> Copula (95% CI) :
#> AIC = -1432.074
#> est low upper se
#> theta -5.1126 -5.4868 -4.7384 0.1909Notice that the print method also can be used to print the results obtained from the fitting process.
timeroc_gofAfter fitting the model with either PH or copula model, its
goodness-of-fit can be examined through the function
parTimeROC::timeroc_gof(). This will return a list of test
statistic and p-values denoting misspecification of model or not.
Kolmogorov-Smirnov testing is performed for model checking. If
p-value < 0.05, we reject the null hypothesis that the
data (biomarker or time-to-event) are following the assumed
distribution.
For copula model, additional testing is needed to check whether the
copula used is able to model the data or not. After using the Rosenblatt
transformation, we conduct an independent testing to check whether the
empirical conditional and cumulative distribution are independent. If
the p-value < 0.05, we reject the null hypothesis which
stated that the conditional and cumulative are independent. Thus, for
p-value < 0.05, the copula failed to provide a good
estimation for the joint distribution.
library(parTimeROC)
# Copula model
rt <- timeroc_obj("normal-weibull-copula",copula="clayton90")
set.seed(1)
rr <- rtimeroc(rt, n=300, censor.rate = 0,
params.x = c(mean=5, sd=1),
params.t = c(shape=1, scale=5),
params.copula = -2.5)
test <- timeroc_obj("normal-weibull-copula",copula="gumbel90")
jj <- timeroc_fit(test, rr$x, rr$t, rr$event)
timeroc_gof(jj)
Fig.2. Residual plots for biomarker and time-to-event distribution when misspecified
#> $ks_x
#>
#> Asymptotic two-sample Kolmogorov-Smirnov test
#>
#> data: df$x and theo.q
#> D = 0.04, p-value = 0.97
#> alternative hypothesis: two-sided
#>
#>
#> $ks_t
#>
#> Asymptotic two-sample Kolmogorov-Smirnov test
#>
#> data: df$t and theo.q
#> D = 0.036667, p-value = 0.9877
#> alternative hypothesis: two-sided
#>
#>
#> $ind_u
#> $ind_u$statistic
#> [1] 1.875196
#>
#> $ind_u$p.value
#> [1] 0.06076572
#>
#>
#> $ind_v
#> $ind_v$statistic
#> [1] 2.674574
#>
#> $ind_v$p.value
#> [1] 0.007482435test <- timeroc_obj("normal-weibull-copula",copula="clayton90")
jj <- timeroc_fit(test, rr$x, rr$t, rr$event)
timeroc_gof(jj)
Fig.3. Residual plots for biomarker and time-to-event distribution when correct specification
#> $ks_x
#>
#> Asymptotic two-sample Kolmogorov-Smirnov test
#>
#> data: df$x and theo.q
#> D = 0.04, p-value = 0.97
#> alternative hypothesis: two-sided
#>
#>
#> $ks_t
#>
#> Asymptotic two-sample Kolmogorov-Smirnov test
#>
#> data: df$t and theo.q
#> D = 0.036667, p-value = 0.9877
#> alternative hypothesis: two-sided
#>
#>
#> $ind_u
#> $ind_u$statistic
#> [1] 1.385664
#>
#> $ind_u$p.value
#> [1] 0.1658495
#>
#>
#> $ind_v
#> $ind_v$statistic
#> [1] 0.07947699
#>
#> $ind_v$p.value
#> [1] 0.9366532library(parTimeROC)
# PH model
rt <- timeroc_obj("normal-weibull-PH")
set.seed(1)
rr <- rtimeroc(rt, n=300, censor.rate = 0,
params.x = c(mean=5, sd=1),
params.t = c(shape=1, scale=5),
params.ph = 1.2)
test <- timeroc_obj("lognormal-lognormal-PH")
jj <- timeroc_fit(test, rr$x, rr$t, rr$event)
timeroc_gof(jj)
Fig.4. Residual plots for biomarker and time-to-event distribution when misspecified
#> $ks_x
#>
#> Asymptotic two-sample Kolmogorov-Smirnov test
#>
#> data: df$x and theo.q
#> D = 0.056667, p-value = 0.7212
#> alternative hypothesis: two-sided
#>
#>
#> $ks_t
#> A p-value F(ym) ym
#> 0.673 0.756 0.996 4.960
#>
#> $df
#> x t event coxsnell mresid sgn devresid
#> 180 6.207908 1.584490e-06 1 2.131038e-05 0.999978690 1 4.417315332
#> 280 5.919804 3.856797e-05 1 2.895444e-03 0.997104556 1 3.113683487
#> 174 4.922847 5.297660e-05 1 1.589046e-03 0.998410954 1 3.300366838
#> 203 6.586588 5.434546e-05 1 9.734000e-03 0.990266000 1 2.698838419
#> 283 6.324259 9.838639e-05 1 1.703462e-02 0.982965384 1 2.485776487
#> 11 6.511781 1.362743e-04 1 3.250970e-02 0.967490303 1 2.217533118
#> 239 6.096777 1.485618e-04 1 2.345591e-02 0.976544085 1 2.356305793
#> 272 5.394379 2.055676e-04 1 1.706850e-02 0.982931497 1 2.484990516
#> 5 5.329508 2.056777e-04 1 1.593797e-02 0.984062028 1 2.511966883
#> 206 7.497662 2.412804e-04 1 1.985675e-01 0.801432458 1 1.276866098
#> 50 5.881108 2.569037e-04 1 3.836344e-02 0.961636556 1 2.144301142
#> 178 7.075245 2.909709e-04 1 1.610414e-01 0.838958564 1 1.405087198
#> 9 5.575781 3.123474e-04 1 3.558197e-02 0.964418029 1 2.177842126
#> 254 4.752336 3.308331e-04 1 1.589561e-02 0.984104390 1 2.513009312
#> 208 5.541327 3.523408e-04 1 3.995380e-02 0.960046204 1 2.126022288
#> 70 7.172612 3.641114e-04 1 2.374854e-01 0.762514566 1 1.162010690
#> 31 6.358680 3.731538e-04 1 1.027467e-01 0.897253341 1 1.660262387
#> 43 5.696963 3.735145e-04 1 5.076778e-02 0.949232218 1 2.015569953
#> 241 5.707311 4.171255e-04 1 5.893075e-02 0.941069251 1 1.944388345
#> 109 5.384185 4.611596e-04 1 4.727337e-02 0.952726631 1 2.048941934
#> 30 5.417942 4.789896e-04 1 5.135475e-02 0.948645251 1 2.010150549
#> 163 6.058483 4.974118e-04 1 1.065678e-01 0.893432185 1 1.640452102
#> 4 6.595281 5.261248e-04 1 2.024376e-01 0.797562396 1 1.264722241
#> 110 6.682176 5.587205e-04 1 2.390025e-01 0.760997508 1 1.157828825
#> 171 7.307978 5.743735e-04 1 4.819979e-01 0.518002066 1 0.650866170
#> 251 5.136222 5.780635e-04 1 4.784186e-02 0.952158143 1 2.043377698
#> 93 6.160403 6.104503e-04 1 1.524681e-01 0.847531905 1 1.437545140
#> 21 5.918977 6.605556e-04 1 1.295785e-01 0.870421494 1 1.531696354
#> 61 7.401618 6.870186e-04 1 6.609741e-01 0.339025946 1 0.387336408
#> 48 5.768533 7.316284e-04 1 1.246976e-01 0.875302357 1 1.553422655
#> 2 5.183643 7.346172e-04 1 6.712258e-02 0.932877423 1 1.880615541
#> 90 5.267099 7.565577e-04 1 7.598775e-02 0.924012252 1 1.818334906
#> 147 7.087167 7.590941e-04 1 5.321981e-01 0.467801939 1 0.570854842
#> 274 7.649167 7.724052e-04 1 9.897290e-01 0.010271029 1 0.010306405
#> 6 4.179532 8.469456e-04 1 2.720088e-02 0.972799120 1 2.294213075
#> 126 5.712666 8.497752e-04 1 1.402481e-01 0.859751873 1 1.486331200
#> 185 5.521023 9.809825e-04 1 1.352010e-01 0.864798955 1 1.507443817
#> 187 6.464587 1.020379e-03 1 3.874071e-01 0.612592866 1 0.819373232
#> 160 6.869291 1.054860e-03 1 6.200787e-01 0.379921277 1 0.442690771
#> 15 6.124931 1.079218e-03 1 2.876596e-01 0.712340410 1 1.033089606
#> 73 5.610726 1.123521e-03 1 1.740439e-01 0.825956094 1 1.358301573
#> 282 4.592471 1.147806e-03 1 6.016727e-02 0.939832728 1 1.934318483
#> 201 5.409402 1.155275e-03 1 1.449571e-01 0.855042885 1 1.467156744
#> 216 6.519745 1.209462e-03 1 4.996520e-01 0.500347997 1 0.622085878
#> 102 5.042116 1.229168e-03 1 1.051468e-01 0.894853152 1 1.647752532
#> 39 6.100025 1.231089e-03 1 3.257711e-01 0.674228898 1 0.945866150
#> 122 6.343039 1.248639e-03 1 4.290984e-01 0.570901616 1 0.741845587
#> 55 6.433024 1.257002e-03 1 4.759558e-01 0.524044242 1 0.660887482
#> 182 5.983896 1.290249e-03 1 3.035914e-01 0.696408574 1 0.995654459
#> 152 4.981440 1.303752e-03 1 1.053863e-01 0.894613737 1 1.646517080
#> 217 4.691259 1.393261e-03 1 8.335384e-02 0.916646162 1 1.770883655
#> 41 4.835476 1.514742e-03 1 1.068080e-01 0.893191958 1 1.639225484
#> 288 5.945185 1.550642e-03 1 3.583808e-01 0.641619199 1 0.876972028
#> 234 5.570508 1.564560e-03 1 2.426722e-01 0.757327756 1 1.147794215
#> 166 7.206102 1.582535e-03 1 1.408195e+00 -0.408194560 -1 -0.363004490
#> 95 6.586833 1.583196e-03 1 7.274998e-01 0.272500198 1 0.302130283
#> 123 4.785421 1.590259e-03 1 1.069137e-01 0.893086310 1 1.638686726
#> 295 6.778429 1.600860e-03 1 9.036812e-01 0.096318810 1 0.099597558
#> 99 3.775387 1.616006e-03 1 3.703913e-02 0.962960868 1 2.160009001
#> 199 5.411975 1.625319e-03 1 2.137973e-01 0.786202688 1 1.230060295
#> 193 4.268252 1.705735e-03 1 6.655795e-02 0.933442050 1 1.884802501
#> 59 5.569720 1.787185e-03 1 2.810998e-01 0.718900152 1 1.048947272
#> 71 5.475510 1.847682e-03 1 2.637217e-01 0.736278310 1 1.092321039
#> 142 6.176583 1.847913e-03 1 5.573840e-01 0.442616032 1 0.532700468
#> 133 5.531496 1.873528e-03 1 2.842750e-01 0.715725041 1 1.041238016
#> 161 5.425100 1.876214e-03 1 2.541607e-01 0.745839252 1 1.117093634
#> 101 4.379633 1.900729e-03 1 8.447587e-02 0.915524133 1 1.763953090
#> 169 4.855600 1.952600e-03 1 1.446040e-01 0.855395959 1 1.468577586
#> 92 6.207868 1.973661e-03 1 6.195337e-01 0.380466273 1 0.443445293
#> 40 5.763176 2.000774e-03 1 3.912341e-01 0.608765855 1 0.812013802
#> 258 5.572740 2.044065e-03 1 3.268376e-01 0.673162446 1 0.943535438
#> 60 4.864945 2.075975e-03 1 1.561746e-01 0.843825421 1 1.423344918
#> 19 5.821221 2.162081e-03 1 4.529615e-01 0.547038533 1 0.699870968
#> 172 5.105802 2.162565e-03 1 2.111395e-01 0.788860544 1 1.238043529
#> 148 5.017396 2.181191e-03 1 1.939273e-01 0.806072739 1 1.291665124
#> 252 5.407168 2.257563e-03 1 3.051441e-01 0.694855920 1 0.992083970
#> 245 6.162965 2.291417e-03 1 6.947238e-01 0.305276158 1 0.343408515
#> 34 4.946195 2.486713e-03 1 2.070655e-01 0.792934507 1 1.250428436
#> 297 5.765599 2.579352e-03 1 5.163304e-01 0.483669648 1 0.595548241
#> 159 3.615573 2.589888e-03 1 5.227349e-02 0.947726505 1 2.001768882
#> 214 3.629792 2.621402e-03 1 5.376348e-02 0.946236518 1 1.988428674
#> 108 5.910174 2.633230e-03 1 6.159373e-01 0.384062664 1 0.448435913
#> 56 6.980400 2.707695e-03 1 1.988415e+00 -0.988414985 -1 -0.775986022
#> 94 5.700214 2.746805e-03 1 5.149560e-01 0.485043970 1 0.597711954
#> 230 5.263176 2.871097e-03 1 3.385427e-01 0.661457276 1 0.918311171
#> 135 5.306558 2.878444e-03 1 3.555474e-01 0.644452605 1 0.882773034
#> 62 4.960760 2.918194e-03 1 2.494177e-01 0.750582322 1 1.129640691
#> 183 5.219925 2.983556e-03 1 3.366834e-01 0.663316639 1 0.922275157
#> 51 5.398106 3.011408e-03 1 4.112154e-01 0.588784572 1 0.774407483
#> 284 4.298768 3.052306e-03 1 1.290339e-01 0.870966073 1 1.534088547
#> 157 6.000029 3.108070e-03 1 8.082057e-01 0.191794317 1 0.205642300
#> 85 5.593946 3.132134e-03 1 5.282332e-01 0.471766790 1 0.576975939
#> 197 6.441158 3.162805e-03 1 1.318130e+00 -0.318129773 -1 -0.289537147
#> 87 6.063100 3.293417e-03 1 9.186709e-01 0.081329149 1 0.083644865
#> 170 5.207538 3.330429e-03 1 3.729790e-01 0.627021042 1 0.847599236
#> 68 6.465555 3.442322e-03 1 1.478304e+00 -0.478303723 -1 -0.418111052
#> 215 5.987838 3.627942e-03 1 9.377089e-01 0.062291095 1 0.063633623
#> 131 5.060160 3.631898e-03 1 3.488022e-01 0.651197840 1 0.896719079
#> 82 4.864821 3.655713e-03 1 2.850911e-01 0.714908868 1 1.039266588
#> 96 5.558486 3.684344e-03 1 6.025747e-01 0.397425290 1 0.467158070
#> 224 4.659031 3.709862e-03 1 2.323901e-01 0.767609932 1 1.176204116
#> 250 6.020464 3.806544e-03 1 1.020473e+00 -0.020473094 -1 -0.020335023
#> 106 6.767287 3.885685e-03 1 2.313026e+00 -1.313025659 -1 -0.974134675
#> 289 5.433702 3.995757e-03 1 5.735310e-01 0.426468971 1 0.508869862
#> 104 5.158029 4.006017e-03 1 4.284770e-01 0.571523011 1 0.742960608
#> 278 5.741001 4.015517e-03 1 8.001913e-01 0.199808718 1 0.214922123
#> 120 4.822670 4.064548e-03 1 3.040620e-01 0.695937964 1 0.994570830
#> 290 6.005159 4.105513e-03 1 1.085223e+00 -0.085222751 -1 -0.082915435
#> 18 5.943836 4.305156e-03 1 1.067111e+00 -0.067111174 -1 -0.065666034
#> 220 4.955291 4.348745e-03 1 3.753921e-01 0.624607871 1 0.842824122
#> 138 4.471720 4.356570e-03 1 2.244638e-01 0.775536244 1 1.198753338
#> 265 6.719627 4.356954e-03 1 2.472257e+00 -1.472256627 -1 -1.065011999
#> 52 4.387974 4.408170e-03 1 2.077521e-01 0.792247885 1 1.248328306
#> 233 4.359518 4.482706e-03 1 2.050095e-01 0.794990454 1 1.256748409
#> 107 5.716707 4.505199e-03 1 8.770923e-01 0.122907662 1 0.128338160
#> 113 6.432282 4.706226e-03 1 1.967727e+00 -0.967726772 -1 -0.762689738
#> 226 6.803142 4.738835e-03 1 2.943720e+00 -1.943720178 -1 -1.314569154
#> 8 5.738325 4.760157e-03 1 9.491309e-01 0.050869148 1 0.051758242
#> 218 3.746710 4.840431e-03 1 1.152274e-01 0.884772614 1 1.597545127
#> 194 5.830373 4.899466e-03 1 1.078088e+00 -0.078088297 -1 -0.076143545
#> 151 5.450187 5.053122e-03 1 7.412365e-01 0.258763514 1 0.285208858
#> 249 4.833879 5.062990e-03 1 3.847246e-01 0.615275443 1 0.824563149
#> 66 5.188792 5.248453e-03 1 5.825742e-01 0.417425806 1 0.495727594
#> 181 3.768677 5.286819e-03 1 1.289132e-01 0.871086830 1 1.534620066
#> 124 4.820443 5.337817e-03 1 3.999172e-01 0.600082782 1 0.795506035
#> 49 4.887654 5.363530e-03 1 4.317297e-01 0.568270340 1 0.737136804
#> 1 4.373546 5.400864e-03 1 2.511521e-01 0.748847850 1 1.125031999
#> 204 4.669092 5.424176e-03 1 3.457752e-01 0.654224781 1 0.903041074
#> 202 6.688873 5.501575e-03 1 3.027711e+00 -2.027711151 -1 -1.356395380
#> 179 6.027392 5.660975e-03 1 1.537759e+00 -0.537758744 -1 -0.463535864
#> 64 5.028002 5.662658e-03 1 5.294170e-01 0.470582959 1 0.575144924
#> 86 5.332950 5.664599e-03 1 7.333142e-01 0.266685838 1 0.294941463
#> 83 6.178087 5.846703e-03 1 1.864996e+00 -0.864995614 -1 -0.695322821
#> 207 5.667066 5.876265e-03 1 1.086409e+00 -0.086409049 -1 -0.084038585
#> 119 5.494188 5.899703e-03 1 9.069424e-01 0.093057615 1 0.096111794
#> 269 4.668967 5.970098e-03 1 3.803613e-01 0.619638652 1 0.833060515
#> 287 4.331821 6.058580e-03 1 2.693125e-01 0.730687479 1 1.078142204
#> 26 4.943871 6.314466e-03 1 5.391229e-01 0.460877098 1 0.560240338
#> 81 4.431331 6.402171e-03 1 3.162470e-01 0.683752971 1 0.966931914
#> 263 4.715669 6.423455e-03 1 4.297703e-01 0.570229669 1 0.740641152
#> 45 4.311244 6.591379e-03 1 2.862872e-01 0.713712825 1 1.036385100
#> 114 4.349304 6.615928e-03 1 2.992452e-01 0.700754756 1 1.005720709
#> 25 5.619826 6.623362e-03 1 1.162505e+00 -0.162504838 -1 -0.154452697
#> 164 5.886423 6.755703e-03 1 1.575385e+00 -0.575384953 -1 -0.491701730
#> 225 3.843428 6.934491e-03 1 1.826221e-01 0.817377939 1 1.328878147
#> 238 3.813541 6.991735e-03 1 1.783151e-01 0.821684920 1 1.343516475
#> 23 5.074565 6.994508e-03 1 6.852397e-01 0.314760315 1 0.355601692
#> 273 4.148143 7.097236e-03 1 2.585961e-01 0.741403883 1 1.105517023
#> 212 5.420695 7.099505e-03 1 1.005974e+00 -0.005974452 -1 -0.005962596
#> 128 4.962366 7.101513e-03 1 6.169793e-01 0.383020688 1 0.446987914
#> 242 6.034108 7.188278e-03 1 1.959580e+00 -0.959580014 -1 -0.757429656
#> 7 5.487429 7.236927e-03 1 1.100596e+00 -0.100596274 -1 -0.097408149
#> 222 5.002132 7.294616e-03 1 6.607639e-01 0.339236086 1 0.387614710
#> 275 5.156012 7.337022e-03 1 7.830996e-01 0.216900406 1 0.234925476
#> 248 4.744329 7.403683e-03 1 5.091166e-01 0.490883380 1 0.606951057
#> 22 5.782136 7.505808e-03 1 1.561765e+00 -0.561764574 -1 -0.481556341
#> 76 5.291446 7.579080e-03 1 9.338333e-01 0.066166663 1 0.067685061
#> 236 4.901821 7.672389e-03 1 6.234807e-01 0.376519322 1 0.437991282
#> 223 4.369700 7.844831e-03 1 3.610064e-01 0.638993633 1 0.871626124
#> 219 5.642241 8.118891e-03 1 1.451164e+00 -0.451163617 -1 -0.396983345
#> 255 5.695551 8.198243e-03 1 1.550561e+00 -0.550561131 -1 -0.473168565
#> 12 5.389843 8.235466e-03 1 1.123788e+00 -0.123787742 -1 -0.119019746
#> 266 5.270055 8.264447e-03 1 9.922663e-01 0.007733713 1 0.007753741
#> 53 5.341120 8.365532e-03 1 1.083020e+00 -0.083020320 -1 -0.080828082
#> 175 4.665999 8.406353e-03 1 5.293532e-01 0.470646811 1 0.575243609
#> 281 5.398130 8.470306e-03 1 1.164784e+00 -0.164784009 -1 -0.156514148
#> 240 4.994656 8.508742e-03 1 7.605343e-01 0.239465737 1 0.261795256
#> 111 4.364264 8.565672e-03 1 3.905753e-01 0.609424684 1 0.813277041
#> 33 5.387672 9.010942e-03 1 1.222016e+00 -0.222016440 -1 -0.207432524
#> 167 4.744973 9.601335e-03 1 6.536791e-01 0.346320908 1 0.397033563
#> 67 3.195041 9.747181e-03 1 1.268132e-01 0.873186839 1 1.543925913
#> 286 3.998928 9.977149e-03 1 3.057256e-01 0.694274382 1 0.990750089
#> 44 5.556663 1.016483e-02 1 1.640487e+00 -0.640486779 -1 -0.539432600
#> 69 5.153253 1.027810e-02 1 1.077759e+00 -0.077758775 -1 -0.075830052
#> 256 6.146228 1.039499e-02 1 3.143288e+00 -2.143287697 -1 -1.412811670
#> 103 4.089078 1.059418e-02 1 3.561478e-01 0.643852233 1 0.881541066
#> 153 4.681932 1.081129e-02 1 6.833968e-01 0.316603202 1 0.357984421
#> 176 4.965274 1.095095e-02 1 9.358722e-01 0.064127763 1 0.065552235
#> 259 5.374724 1.102584e-02 1 1.458020e+00 -0.458020081 -1 -0.402344813
#> 229 5.197193 1.159346e-02 1 1.264136e+00 -0.264136318 -1 -0.243914663
#> 105 4.345415 1.179439e-02 1 5.175169e-01 0.482483087 1 0.593683393
#> 16 4.955066 1.179744e-02 1 9.922079e-01 0.007792079 1 0.007812411
#> 235 4.940277 1.217200e-02 1 1.005367e+00 -0.005367369 -1 -0.005357796
#> 298 5.955137 1.295442e-02 1 3.145332e+00 -2.145331705 -1 -1.413797970
#> 115 4.792619 1.311679e-02 1 9.200434e-01 0.079956611 1 0.082192908
#> 243 5.223480 1.357681e-02 1 1.503959e+00 -0.503959386 -1 -0.437854233
#> 97 3.723408 1.358351e-02 1 3.034915e-01 0.696508542 1 0.995884805
#> 209 4.986600 1.363832e-02 1 1.172838e+00 -0.172838127 -1 -0.163777691
#> 79 5.074341 1.371792e-02 1 1.294843e+00 -0.294843368 -1 -0.270013450
#> 32 4.897212 1.390672e-02 1 1.085278e+00 -0.085277858 -1 -0.082967626
#> 276 6.130207 1.430250e-02 1 4.150937e+00 -3.150936842 -1 -1.858818328
#> 231 4.014173 1.444489e-02 1 4.378003e-01 0.562199660 1 0.726350683
#> 271 4.741067 1.461142e-02 1 9.609866e-01 0.039013396 1 0.039532613
#> 267 4.577816 1.467212e-02 1 8.103740e-01 0.189626028 1 0.203142394
#> 127 4.926436 1.486152e-02 1 1.189394e+00 -0.189394126 -1 -0.178606195
#> 264 5.857410 1.489175e-02 1 3.218355e+00 -2.218355054 -1 -1.448782025
#> 132 4.411106 1.510442e-02 1 6.963852e-01 0.303614762 1 0.341284360
#> 88 4.695816 1.523078e-02 1 9.507970e-01 0.049203018 1 0.050033982
#> 261 5.951013 1.529210e-02 1 3.642935e+00 -2.642935234 -1 -1.643256215
#> 63 5.689739 1.539405e-02 1 2.773061e+00 -1.773061389 -1 -1.227281129
#> 211 4.835624 1.569447e-02 1 1.134106e+00 -0.134105652 -1 -0.128540137
#> 29 4.521850 1.588777e-02 1 8.203718e-01 0.179628190 1 0.191673810
#> 237 5.560821 1.626491e-02 1 2.539465e+00 -1.539464838 -1 -1.102280792
#> 112 4.538355 1.627771e-02 1 8.533653e-01 0.146634740 1 0.154485448
#> 189 4.569788 1.630243e-02 1 8.836826e-01 0.116317369 1 0.121160443
#> 100 4.526599 1.635253e-02 1 8.462043e-01 0.153795656 1 0.162473082
#> 13 4.378759 1.643230e-02 1 7.258535e-01 0.274146486 1 0.304172878
#> 116 4.607192 1.669169e-02 1 9.393543e-01 0.060645657 1 0.061916910
#> 177 5.787640 1.715414e-02 1 3.393063e+00 -2.393062799 -1 -1.530574928
#> 20 5.593901 1.736921e-02 1 2.790149e+00 -1.790149154 -1 -1.236166736
#> 285 4.419386 1.760827e-02 1 8.063687e-01 0.193631348 1 0.207763849
#> 279 3.683755 1.802936e-02 1 3.755833e-01 0.624416716 1 0.842446819
#> 125 4.899809 1.869249e-02 1 1.419954e+00 -0.419954152 -1 -0.372369624
#> 72 4.290054 1.887332e-02 1 7.470585e-01 0.252941461 1 0.278101674
#> 42 4.746638 2.012913e-02 1 1.287258e+00 -0.287258479 -1 -0.263604750
#> 27 4.844204 2.036010e-02 1 1.442921e+00 -0.442920569 -1 -0.390515906
#> 196 3.952016 2.046926e-02 1 5.594364e-01 0.440563577 1 0.529644810
#> 213 4.599753 2.063958e-02 1 1.124957e+00 -0.124957180 -1 -0.120101684
#> 200 4.618924 2.065233e-02 1 1.148833e+00 -0.148833038 -1 -0.142030704
#> 77 4.556708 2.141979e-02 1 1.109920e+00 -0.109920166 -1 -0.106132708
#> 292 5.376370 2.146851e-02 1 2.667279e+00 -1.667278793 -1 -1.171511860
#> 192 5.402012 2.179250e-02 1 2.777357e+00 -1.777357283 -1 -1.229518137
#> 168 3.575505 2.181733e-02 1 3.957989e-01 0.604201101 1 0.803303089
#> 137 4.699024 2.242039e-02 1 1.344436e+00 -0.344435630 -1 -0.311323982
#> 296 5.134448 2.242761e-02 1 2.140331e+00 -1.140331427 -1 -0.871057676
#> 293 5.244165 2.268948e-02 1 2.430619e+00 -1.430619293 -1 -1.041607618
#> 58 3.955865 2.303471e-02 1 6.227020e-01 0.377297951 1 0.439065315
#> 210 5.510108 2.319029e-02 1 3.290097e+00 -2.290096855 -1 -1.482686650
#> 36 4.585005 2.350073e-02 1 1.240006e+00 -0.240005945 -1 -0.223113295
#> 129 4.318340 2.352118e-02 1 9.335781e-01 0.066421879 1 0.067952254
#> 17 4.983810 2.374099e-02 1 1.914658e+00 -0.914658235 -1 -0.728174561
#> 268 3.810887 2.376964e-02 1 5.481412e-01 0.451858781 1 0.546559321
#> 74 4.065902 2.434690e-02 1 7.346658e-01 0.265334164 1 0.293275857
#> 244 4.121292 2.538012e-02 1 8.077814e-01 0.192218553 1 0.206131950
#> 78 5.001105 2.589449e-02 1 2.101658e+00 -1.101658403 -1 -0.847268144
#> 262 4.610763 2.628633e-02 1 1.403531e+00 -0.403530615 -1 -0.359276175
#> 191 4.822896 2.648582e-02 1 1.771551e+00 -0.771550879 -1 -0.631973913
#> 186 4.841245 2.697997e-02 1 1.835330e+00 -0.835329577 -1 -0.675433943
#> 198 3.984153 2.773864e-02 1 7.528528e-01 0.247147192 1 0.271065869
#> 47 5.364582 2.786027e-02 1 3.297290e+00 -2.297289999 -1 -1.486061291
#> 227 4.668868 2.791929e-02 1 1.572078e+00 -0.572077873 -1 -0.489243585
#> 144 4.536470 2.819286e-02 1 1.376275e+00 -0.376275082 -1 -0.337326098
#> 173 5.456999 2.934729e-02 1 3.803133e+00 -2.803132652 -1 -1.713071827
#> 146 4.249181 2.939380e-02 1 1.049295e+00 -0.049294947 -1 -0.048507472
#> 260 4.574732 2.990610e-02 1 1.507058e+00 -0.507058160 -1 -0.440224143
#> 162 4.761353 3.042858e-02 1 1.866257e+00 -0.866256640 -1 -0.696163791
#> 65 4.256727 3.117679e-02 1 1.111610e+00 -0.111610072 -1 -0.107708795
#> 57 4.632779 3.160045e-02 1 1.679481e+00 -0.679480818 -1 -0.567443582
#> 80 4.410479 3.230765e-02 1 1.349568e+00 -0.349568299 -1 -0.315542505
#> 300 4.694185 3.260501e-02 1 1.840865e+00 -0.840865362 -1 -0.679160652
#> 28 3.529248 3.288957e-02 1 5.348434e-01 0.465156576 1 0.566788610
#> 118 4.720887 3.326862e-02 1 1.926233e+00 -0.926232915 -1 -0.735753596
#> 195 3.791917 3.474907e-02 1 7.410942e-01 0.258905847 1 0.285383086
#> 150 3.359394 3.477969e-02 1 4.674296e-01 0.532570427 1 0.675183177
#> 89 5.370019 3.532239e-02 1 4.047849e+00 -3.047848749 -1 -1.816404791
#> 38 4.940687 3.639618e-02 1 2.624194e+00 -1.624194265 -1 -1.148407909
#> 232 2.111079 3.741333e-02 1 1.310235e-01 0.868976483 1 1.525386490
#> 188 4.233918 3.822148e-02 1 1.285076e+00 -0.285076132 -1 -0.261756256
#> 75 3.746367 3.847363e-02 1 7.678837e-01 0.232116345 1 0.252984993
#> 158 4.378733 3.894701e-02 1 1.523210e+00 -0.523209554 -1 -0.452526014
#> 10 4.694612 4.001841e-02 1 2.181864e+00 -1.181864416 -1 -0.896308728
#> 253 4.930345 4.015504e-02 1 2.813855e+00 -1.813854676 -1 -1.248438514
#> 140 4.943103 4.185950e-02 1 2.950870e+00 -1.950869643 -1 -1.318157595
#> 46 4.292505 4.325768e-02 1 1.513548e+00 -0.513548410 -1 -0.445177662
#> 98 4.426735 4.402701e-02 1 1.771817e+00 -0.771816547 -1 -0.632156988
#> 54 3.870637 4.496208e-02 1 9.955218e-01 0.004478158 1 0.004484860
#> 165 4.380757 4.498094e-02 1 1.716483e+00 -0.716482512 -1 -0.593641923
#> 143 3.335028 4.525564e-02 1 5.650388e-01 0.434961240 1 0.521343857
#> 37 4.605710 4.624488e-02 1 2.231599e+00 -1.231599363 -1 -0.926154233
#> 84 3.476433 5.110284e-02 1 7.243647e-01 0.275635287 1 0.306022807
#> 35 3.622940 5.139426e-02 1 8.508073e-01 0.149192697 1 0.157333499
#> 184 3.532750 5.404778e-02 1 8.042562e-01 0.195743811 1 0.210207548
#> 130 4.675730 5.421549e-02 1 2.730441e+00 -1.730441387 -1 -1.204971461
#> 91 4.457480 6.019117e-02 1 2.348705e+00 -1.348705062 -1 -0.994827546
#> 155 3.512540 6.029946e-02 1 8.579429e-01 0.142057139 1 0.149403067
#> 299 4.949434 6.122222e-02 1 4.023643e+00 -3.023643374 -1 -1.806352994
#> 3 4.164371 6.466188e-02 1 1.816506e+00 -0.816505980 -1 -0.662708255
#> 117 4.680007 6.538887e-02 1 3.176903e+00 -2.176903155 -1 -1.428983053
#> 139 4.347905 6.637455e-02 1 2.254816e+00 -1.254815843 -1 -0.939944189
#> 156 3.924808 7.424849e-02 1 1.564880e+00 -0.564880024 -1 -0.483881938
#> 145 3.884080 7.488925e-02 1 1.508194e+00 -0.508193644 -1 -0.441091763
#> 247 4.455209 7.715725e-02 1 2.838330e+00 -1.838329520 -1 -1.261042297
#> 154 4.070638 7.754922e-02 1 1.890120e+00 -0.890119557 -1 -0.712010494
#> 228 3.394487 8.530838e-02 1 9.873124e-01 0.012687573 1 0.012741632
#> 136 3.463550 9.491171e-02 1 1.151214e+00 -0.151213737 -1 -0.144200747
#> 270 4.060171 9.827059e-02 1 2.233113e+00 -1.233113128 -1 -0.927056087
#> 221 3.266782 1.044329e-01 1 1.001592e+00 -0.001591895 -1 -0.001591051
#> 14 2.785300 1.093166e-01 1 6.195863e-01 0.380413684 1 0.443372466
#> 246 2.999835 1.137469e-01 1 8.019561e-01 0.198043922 1 0.212873278
#> 190 4.073891 1.162474e-01 1 2.563631e+00 -1.563630915 -1 -1.115532461
#> 121 4.494043 1.215979e-01 1 4.147420e+00 -3.147419742 -1 -1.857381679
#> 149 3.713699 1.223064e-01 1 1.810979e+00 -0.810979208 -1 -0.658956008
#> 141 3.085641 1.298620e-01 1 9.673277e-01 0.032672339 1 0.033035105
#> 291 4.609881 1.313113e-01 1 4.960460e+00 -3.960459651 -1 -2.172077919
#> 134 3.481606 2.284535e-01 1 2.186512e+00 -1.186511657 -1 -0.899115381
#> 205 2.714764 2.495848e-01 1 1.023410e+00 -0.023410062 -1 -0.023229840
#> 277 2.710876 2.614048e-01 1 1.050993e+00 -0.050992746 -1 -0.050150895
#> 24 3.010648 2.686244e-01 1 1.473576e+00 -0.473576330 -1 -0.414449047
#> 294 3.573743 2.974617e-01 1 2.873731e+00 -1.873731125 -1 -1.279155891
#> 257 2.596904 2.979228e-01 1 1.014189e+00 -0.014188826 -1 -0.014122269
#> Hcoxsnell
#> 180 0.003333333
#> 280 0.010033520
#> 174 0.006677815
#> 203 0.013400523
#> 283 0.023570092
#> 11 0.037292421
#> 239 0.030407719
#> 272 0.026983062
#> 5 0.020168732
#> 206 0.194036953
#> 50 0.047709172
#> 178 0.174036313
#> 9 0.040752629
#> 254 0.016778901
#> 208 0.051205675
#> 70 0.231078173
#> 31 0.101485511
#> 43 0.061769143
#> 241 0.076029619
#> 109 0.054714447
#> 30 0.065315242
#> 163 0.112596723
#> 4 0.198085536
#> 110 0.235279853
#> 171 0.466349036
#> 251 0.058235574
#> 93 0.166146736
#> 21 0.142844740
#> 61 0.627152655
#> 48 0.127606369
#> 2 0.086860038
#> 90 0.090496401
#> 147 0.515271714
#> 274 0.956174562
#> 6 0.033844145
#> 126 0.154427867
#> 185 0.150551898
#> 187 0.394707313
#> 160 0.596475499
#> 15 0.287127237
#> 73 0.178004567
#> 282 0.079626741
#> 201 0.162225167
#> 216 0.471668185
#> 102 0.105175548
#> 39 0.323248484
#> 122 0.429886746
#> 55 0.461058031
#> 182 0.300520272
#> 152 0.108879251
#> 217 0.094146037
#> 41 0.116328067
#> 288 0.360723527
#> 234 0.239499263
#> 166 1.408155097
#> 95 0.671745880
#> 123 0.120073385
#> 295 0.841764850
#> 99 0.044224851
#> 199 0.218578028
#> 193 0.083236849
#> 59 0.269544395
#> 71 0.260867564
#> 142 0.537807820
#> 133 0.273911207
#> 161 0.252265374
#> 101 0.097809040
#> 169 0.158318917
#> 92 0.584463379
#> 40 0.404632932
#> 258 0.327856779
#> 60 0.170083744
#> 19 0.450559271
#> 172 0.214445796
#> 148 0.190004695
#> 252 0.309549664
#> 245 0.646021077
#> 34 0.206232209
#> 297 0.487797528
#> 159 0.068873961
#> 214 0.072445389
#> 108 0.572593838
#> 56 2.029120841
#> 94 0.482392122
#> 230 0.337137571
#> 135 0.351222285
#> 62 0.243736551
#> 183 0.332486409
#> 51 0.419708563
#> 284 0.139013323
#> 157 0.774575718
#> 85 0.498696791
#> 197 1.342342829
#> 87 0.857329288
#> 170 0.370315908
#> 68 1.493311473
#> 215 0.897334409
#> 131 0.346505304
#> 82 0.278297172
#> 96 0.566711485
#> 224 0.226894072
#> 250 1.027953762
#> 106 2.255418685
#> 289 0.555049579
#> 104 0.424784705
#> 278 0.739111157
#> 120 0.305024777
#> 290 1.105286363
#> 18 1.065872713
#> 220 0.375146826
#> 138 0.222727405
#> 265 2.355492841
#> 52 0.210330570
#> 233 0.202150577
#> 107 0.826438954
#> 113 2.004120841
#> 226 2.901934887
#> 8 0.913795593
#> 218 0.123832784
#> 194 1.085385373
#> 151 0.698149958
#> 249 0.389781204
#> 66 0.560863532
#> 181 0.135196529
#> 124 0.414658058
#> 49 0.440169590
#> 1 0.247991870
#> 204 0.341810469
#> 202 3.031101554
#> 179 1.586400025
#> 64 0.509716159
#> 86 0.678281828
#> 83 1.865976634
#> 207 1.125591454
#> 119 0.849516788
#> 269 0.384879244
#> 287 0.265196569
#> 26 0.526476284
#> 81 0.318661328
#> 263 0.435014951
#> 45 0.282702458
#> 114 0.291571682
#> 25 1.222440344
#> 164 1.671194526
#> 225 0.185988631
#> 238 0.181988631
#> 23 0.639691964
#> 273 0.256557220
#> 212 1.009520191
#> 128 0.578510998
#> 242 1.979730597
#> 7 1.135900733
#> 222 0.620941475
#> 275 0.732166712
#> 248 0.477015778
#> 22 1.619460134
#> 76 0.881139811
#> 236 0.608633666
#> 223 0.365508216
#> 219 1.449827123
#> 255 1.602793467
#> 12 1.167482013
#> 266 0.973642144
#> 53 1.095286363
#> 175 0.504191297
#> 281 1.233803980
#> 240 0.718420846
#> 111 0.399657808
#> 33 1.268690846
#> 167 0.614768636
#> 67 0.131394247
#> 286 0.314095118
#> 44 1.689051668
#> 69 1.075581451
#> 256 3.102530125
#> 103 0.355961622
#> 153 0.633402655
#> 176 0.889204328
#> 259 1.464112837
#> 229 1.292643800
#> 105 0.493232310
#> 16 0.964870214
#> 235 1.000429282
#> 298 3.179453202
#> 115 0.865203303
#> 243 1.508236846
#> 97 0.296035967
#> 209 1.245298233
#> 79 1.329684601
#> 32 1.115387373
#> 276 5.282663880
#> 231 0.445350937
#> 271 0.930532288
#> 267 0.781822094
#> 127 1.256926140
#> 264 3.353695626
#> 132 0.652390504
#> 88 0.922128926
#> 261 3.832663880
#> 63 2.591850630
#> 211 1.189104266
#> 29 0.789121365
#> 237 2.391207127
#> 112 0.811344400
#> 189 0.834072542
#> 100 0.796474306
#> 13 0.665252374
#> 116 0.905531130
#> 177 3.689806737
#> 20 2.684924223
#> 285 0.760238616
#> 279 0.380001195
#> 125 1.421853727
#> 72 0.704861368
#> 42 1.317184601
#> 27 1.435742616
#> 196 0.543522105
#> 213 1.178234701
#> 200 1.200093277
#> 77 1.146317399
#> 292 2.506705703
#> 192 2.637305176
#> 168 0.409632932
#> 137 1.355163342
#> 296 2.081077656
#> 293 2.321010083
#> 58 0.602536105
#> 210 3.453695626
#> 36 1.280595608
#> 129 0.873139811
#> 17 1.932665260
#> 268 0.532126002
#> 74 0.684860775
#> 244 0.767381473
#> 78 2.054761867
#> 262 1.394641583
#> 191 1.744619930
#> 186 1.823866705
#> 198 0.711618124
#> 47 3.564806737
#> 227 1.653650666
#> 144 1.381308250
#> 173 3.999330547
#> 146 1.046733519
#> 260 1.523388361
#> 162 1.887715765
#> 65 1.156843715
#> 57 1.707233487
#> 80 1.368150355
#> 300 1.844700039
#> 28 0.520858306
#> 118 1.955921074
#> 195 0.691483292
#> 150 0.455794873
#> 89 4.449330547
#> 38 2.466705703
#> 232 0.146690894
#> 188 1.304838922
#> 75 0.725270161
#> 158 1.570270992
#> 10 2.108104684
#> 253 2.734924223
#> 140 2.964434887
#> 46 1.554397977
#> 98 1.763850699
#> 54 0.982491702
#> 165 1.725752005
#> 143 0.549269232
#> 37 2.164453890
#> 84 0.658800761
#> 35 0.803881713
#> 184 0.753146417
#> 130 2.548372369
#> 91 2.287676749
#> 155 0.818863197
#> 299 4.199330547
#> 3 1.803458542
#> 117 3.262786535
#> 139 2.224168685
#> 156 1.636409287
#> 145 1.538772977
#> 247 2.787555802
#> 154 1.909937987
#> 228 0.947553872
#> 136 1.211204389
#> 270 2.193865655
#> 221 0.991420273
#> 14 0.590451403
#> 246 0.746104164
#> 190 2.428244164
#> 121 4.782663880
#> 149 1.783458542
#> 141 0.939006864
#> 291 6.282663880
#> 134 2.135882461
#> 205 1.037299557
#> 277 1.056257329
#> 24 1.478605591
#> 294 2.843111358
#> 257 1.018694503test <- timeroc_obj("normal-weibull-PH")
jj <- timeroc_fit(test, rr$x, rr$t, rr$event)
timeroc_gof(jj)
Fig.5. Residual plots for biomarker and time-to-event distribution when correct specification
#> $ks_x
#>
#> Asymptotic two-sample Kolmogorov-Smirnov test
#>
#> data: df$x and theo.q
#> D = 0.04, p-value = 0.97
#> alternative hypothesis: two-sided
#>
#>
#> $ks_t
#> A p-value F(ym) ym
#> 0.500 0.964 0.998 6.353
#>
#> $df
#> x t event coxsnell mresid sgn devresid
#> 180 6.207908 1.584490e-06 1 0.0006242926 0.999375707 1 3.571978642
#> 280 5.919804 3.856797e-05 1 0.0100663998 0.989933600 1 2.686491598
#> 174 4.922847 5.297660e-05 1 0.0045542511 0.995445749 1 2.965214469
#> 203 6.586588 5.434546e-05 1 0.0293308021 0.970669198 1 2.262055637
#> 283 6.324259 9.838639e-05 1 0.0390563095 0.960943690 1 2.136261758
#> 11 6.511781 1.362743e-04 1 0.0659149313 0.934085069 1 1.889605895
#> 239 6.096777 1.485618e-04 1 0.0453196568 0.954680343 1 2.068494173
#> 272 5.394379 2.055676e-04 1 0.0285914110 0.971408589 1 2.272989368
#> 5 5.329508 2.056777e-04 1 0.0266281163 0.973371884 1 2.303221975
#> 206 7.497662 2.412804e-04 1 0.3410192506 0.658980749 1 0.913055968
#> 50 5.881108 2.569037e-04 1 0.0607733923 0.939226608 1 1.929443757
#> 178 7.075245 2.909709e-04 1 0.2564927259 0.743507274 1 1.110988478
#> 9 5.575781 3.123474e-04 1 0.0524346020 0.947565398 1 2.000311557
#> 254 4.752336 3.308331e-04 1 0.0223265966 0.977673403 1 2.376679715
#> 208 5.541327 3.523408e-04 1 0.0567388301 0.943261170 1 1.962669253
#> 70 7.172612 3.641114e-04 1 0.3550710578 0.644928942 1 0.883751554
#> 31 6.358680 3.731538e-04 1 0.1479741801 0.852025820 1 1.455123128
#> 43 5.696963 3.735145e-04 1 0.0713092457 0.928690754 1 1.850426184
#> 241 5.707311 4.171255e-04 1 0.0802903292 0.919709671 1 1.790193525
#> 109 5.384185 4.611596e-04 1 0.0619378497 0.938062150 1 1.920188360
#> 30 5.417942 4.789896e-04 1 0.0667024731 0.933297527 1 1.883728070
#> 163 6.058483 4.974118e-04 1 0.1403914855 0.859608514 1 1.485740165
#> 4 6.595281 5.261248e-04 1 0.2682371474 0.731762853 1 1.080852403
#> 110 6.682176 5.587205e-04 1 0.3130012559 0.686998744 1 0.974216949
#> 171 7.307978 5.743735e-04 1 0.6418188502 0.358181150 1 0.412960119
#> 251 5.136222 5.780635e-04 1 0.0586472529 0.941352747 1 1.946721242
#> 93 6.160403 6.104503e-04 1 0.1916735999 0.808326400 1 1.298949540
#> 21 5.918977 6.605556e-04 1 0.1584883504 0.841511650 1 1.414611281
#> 61 7.401618 6.870186e-04 1 0.8468933517 0.153106648 1 0.161702548
#> 48 5.768533 7.316284e-04 1 0.1482191456 0.851780854 1 1.454154416
#> 2 5.183643 7.346172e-04 1 0.0779887186 0.922011281 1 1.805092693
#> 90 5.267099 7.565577e-04 1 0.0879983414 0.912001659 1 1.742662132
#> 147 7.087167 7.590941e-04 1 0.6592240048 0.340775995 1 0.389655979
#> 274 7.649167 7.724052e-04 1 1.2473022318 -0.247302232 -1 -0.229430719
#> 6 4.179532 8.469456e-04 1 0.0295330797 0.970466920 1 2.259104857
#> 126 5.712666 8.497752e-04 1 0.1611428213 0.838857179 1 1.404711387
#> 185 5.521023 9.809825e-04 1 0.1499002754 0.850099725 1 1.447539507
#> 187 6.464587 1.020379e-03 1 0.4416325977 0.558367402 1 0.719596508
#> 160 6.869291 1.054860e-03 1 0.7132159412 0.286784059 1 0.319959319
#> 15 6.124931 1.079218e-03 1 0.3204427849 0.679557215 1 0.957595240
#> 73 5.610726 1.123521e-03 1 0.1888118368 0.811188163 1 1.308293674
#> 282 4.592471 1.147806e-03 1 0.0625969205 0.937403080 1 1.915012332
#> 201 5.409402 1.155275e-03 1 0.1553088320 0.844691168 1 1.426638365
#> 216 6.519745 1.209462e-03 1 0.5535903284 0.446409672 1 0.538369155
#> 102 5.042116 1.229168e-03 1 0.1099343513 0.890065649 1 1.623456959
#> 39 6.100025 1.231089e-03 1 0.3542472714 0.645752729 1 0.885446073
#> 122 6.343039 1.248639e-03 1 0.4697377044 0.530262296 1 0.671295047
#> 55 6.433024 1.257002e-03 1 0.5222016079 0.477798392 1 0.586349983
#> 182 5.983896 1.290249e-03 1 0.3261232861 0.673876714 1 0.945095844
#> 152 4.981440 1.303752e-03 1 0.1088576600 0.891142340 1 1.628847311
#> 217 4.691259 1.393261e-03 1 0.0842639892 0.915736011 1 1.765256167
#> 41 4.835476 1.514742e-03 1 0.1071691073 0.892830893 1 1.637385940
#> 288 5.945185 1.550642e-03 1 0.3735154255 0.626484574 1 0.846535765
#> 234 5.570508 1.564560e-03 1 0.2490750774 0.750924923 1 1.130553837
#> 166 7.206102 1.582535e-03 1 1.5338952361 -0.533895236 -1 -0.460618780
#> 95 6.586833 1.583196e-03 1 0.7742611608 0.225738839 1 0.245386251
#> 123 4.785421 1.590259e-03 1 0.1063079665 0.893692034 1 1.641781361
#> 295 6.778429 1.600860e-03 1 0.9671188328 0.032881167 1 0.033248632
#> 99 3.775387 1.616006e-03 1 0.0353825975 0.964617403 1 2.180329229
#> 199 5.411975 1.625319e-03 1 0.2169392175 0.783060782 1 1.220718875
#> 193 4.268252 1.705735e-03 1 0.0642708317 0.935729168 1 1.902062150
#> 59 5.569720 1.787185e-03 1 0.2831625415 0.716837458 1 1.043931737
#> 71 5.475510 1.847682e-03 1 0.2635567035 0.736443296 1 1.092742828
#> 142 6.176583 1.847913e-03 1 0.5718148707 0.428185129 1 0.511380217
#> 133 5.531496 1.873528e-03 1 0.2841769968 0.715823003 1 1.041474919
#> 161 5.425100 1.876214e-03 1 0.2530178205 0.746982180 1 1.120101046
#> 101 4.379633 1.900729e-03 1 0.0807380084 0.919261992 1 1.787335357
#> 169 4.855600 1.952600e-03 1 0.1402057394 0.859794261 1 1.486506043
#> 92 6.207868 1.973661e-03 1 0.6309815511 0.369018449 1 0.427691958
#> 40 5.763176 2.000774e-03 1 0.3912322903 0.608767710 1 0.812017355
#> 258 5.572740 2.044065e-03 1 0.3236683800 0.676331620 1 0.950477970
#> 60 4.864945 2.075975e-03 1 0.1503410982 0.849658902 1 1.445814426
#> 19 5.821221 2.162081e-03 1 0.4497464529 0.550253547 1 0.705432840
#> 172 5.105802 2.162565e-03 1 0.2041041588 0.795895841 1 1.259546737
#> 148 5.017396 2.181191e-03 1 0.1866618154 0.813338185 1 1.315384802
#> 252 5.407168 2.257563e-03 1 0.2968589339 0.703141066 1 1.011293387
#> 245 6.162965 2.291417e-03 1 0.6940746064 0.305925394 1 0.344239535
#> 34 4.946195 2.486713e-03 1 0.1959571649 0.804042835 1 1.285158632
#> 297 5.765599 2.579352e-03 1 0.5019681100 0.498031890 1 0.618363642
#> 159 3.615573 2.589888e-03 1 0.0468764921 0.953123508 1 2.052859204
#> 214 3.629792 2.621402e-03 1 0.0481809121 0.951819088 1 2.040084931
#> 108 5.910174 2.633230e-03 1 0.6008259372 0.399174063 1 0.469629525
#> 56 6.980400 2.707695e-03 1 2.0133894792 -1.013389479 -1 -0.791921548
#> 94 5.700214 2.746805e-03 1 0.4963923105 0.503607689 1 0.627345243
#> 230 5.263176 2.871097e-03 1 0.3197568905 0.680243110 1 0.959115405
#> 135 5.306558 2.878444e-03 1 0.3362863597 0.663713640 1 0.923123589
#> 62 4.960760 2.918194e-03 1 0.2325948314 0.767405169 1 1.175629271
#> 183 5.219925 2.983556e-03 1 0.3164231046 0.683576895 1 0.966538284
#> 51 5.398106 3.011408e-03 1 0.3887449204 0.611255080 1 0.816794714
#> 284 4.298768 3.052306e-03 1 0.1169398170 0.883060183 1 1.589361934
#> 157 6.000029 3.108070e-03 1 0.7793748618 0.220625138 1 0.239324055
#> 85 5.593946 3.132134e-03 1 0.5014022264 0.498597774 1 0.619271959
#> 197 6.441158 3.162805e-03 1 1.2904169829 -0.290416983 -1 -0.266276451
#> 87 6.063100 3.293417e-03 1 0.8839389127 0.116061087 1 0.120882048
#> 170 5.207538 3.330429e-03 1 0.3472896195 0.652710380 1 0.899873134
#> 68 6.465555 3.442322e-03 1 1.4392465195 -0.439246519 -1 -0.387625575
#> 215 5.987838 3.627942e-03 1 0.8935085817 0.106491418 1 0.110525305
#> 131 5.060160 3.631898e-03 1 0.3210110602 0.678988940 1 0.956337557
#> 82 4.864821 3.655713e-03 1 0.2603541740 0.739645826 1 1.100969156
#> 96 5.558486 3.684344e-03 1 0.5644529715 0.435547029 1 0.522209116
#> 224 4.659031 3.709862e-03 1 0.2103965773 0.789603423 1 1.240288388
#> 250 6.020464 3.806544e-03 1 0.9705303261 0.029469674 1 0.029764242
#> 106 6.767287 3.885685e-03 1 2.2592284777 -1.259228478 -1 -0.942555149
#> 289 5.433702 3.995757e-03 1 0.5320720140 0.467927986 1 0.571048944
#> 104 5.158029 4.006017e-03 1 0.3933705155 0.606629485 1 0.807928016
#> 278 5.741001 4.015517e-03 1 0.7507091108 0.249290889 1 0.273664572
#> 120 4.822670 4.064548e-03 1 0.2754441100 0.724555890 1 1.062840203
#> 290 6.005159 4.105513e-03 1 1.0269302934 -0.026930293 -1 -0.026692275
#> 18 5.943836 4.305156e-03 1 1.0049380354 -0.004938035 -1 -0.004929931
#> 220 4.955291 4.348745e-03 1 0.3405239993 0.659476001 1 0.914104669
#> 138 4.471720 4.356570e-03 1 0.1999495575 0.800050443 1 1.272509108
#> 265 6.719627 4.356954e-03 1 2.3952676792 -1.395267679 -1 -1.021540688
#> 52 4.387974 4.408170e-03 1 0.1843776172 0.815622383 1 1.322986753
#> 233 4.359518 4.482706e-03 1 0.1816041257 0.818395874 1 1.332313942
#> 107 5.716707 4.505199e-03 1 0.8171935105 0.182806489 1 0.195309333
#> 113 6.432282 4.706226e-03 1 1.8793735323 -0.879373532 -1 -0.704890117
#> 226 6.803142 4.738835e-03 1 2.8499541154 -1.849954115 -1 -1.267005305
#> 8 5.738325 4.760157e-03 1 0.8828803601 0.117119640 1 0.122032292
#> 218 3.746710 4.840431e-03 1 0.0994277219 0.900572278 1 1.677946384
#> 194 5.830373 4.899466e-03 1 1.0051268100 -0.005126810 -1 -0.005118075
#> 151 5.450187 5.053122e-03 1 0.6805354761 0.319464524 1 0.361692693
#> 249 4.833879 5.062990e-03 1 0.3451513381 0.654848662 1 0.904349087
#> 66 5.188792 5.248453e-03 1 0.5289804093 0.471019591 1 0.575819921
#> 181 3.768677 5.286819e-03 1 0.1109769683 0.889023032 1 1.618276590
#> 124 4.820443 5.337817e-03 1 0.3579709032 0.642029097 1 0.877809176
#> 49 4.887654 5.363530e-03 1 0.3873616781 0.612638322 1 0.819460959
#> 1 4.373546 5.400864e-03 1 0.2210070731 0.778992927 1 1.208774293
#> 204 4.669092 5.424176e-03 1 0.3076134161 0.692386584 1 0.986432809
#> 202 6.688873 5.501575e-03 1 2.9035870749 -1.903587075 -1 -1.294326218
#> 179 6.027392 5.660975e-03 1 1.4376095405 -0.437609541 -1 -0.386336246
#> 64 5.028002 5.662658e-03 1 0.4767878918 0.523212108 1 0.659502009
#> 86 5.332950 5.664599e-03 1 0.6679801483 0.332019852 1 0.378092508
#> 83 6.178087 5.846703e-03 1 1.7520358614 -0.752035861 -1 -0.618477809
#> 207 5.667066 5.876265e-03 1 1.0011872619 -0.001187262 -1 -0.001186792
#> 119 5.494188 5.899703e-03 1 0.8303440452 0.169655955 1 0.180328572
#> 269 4.668967 5.970098e-03 1 0.3375763942 0.662423606 1 0.920369312
#> 287 4.331821 6.058580e-03 1 0.2359576648 0.764042335 1 1.166242291
#> 26 4.943871 6.314466e-03 1 0.4829409855 0.517059015 1 0.649310098
#> 81 4.431331 6.402171e-03 1 0.2778602943 0.722139706 1 1.056879486
#> 263 4.715669 6.423455e-03 1 0.3816209096 0.618379090 1 0.830600332
#> 45 4.311244 6.591379e-03 1 0.2503196760 0.749680324 1 1.127241012
#> 114 4.349304 6.615928e-03 1 0.2620115742 0.737988426 1 1.096702489
#> 25 5.619826 6.623362e-03 1 1.0673478272 -0.067347827 -1 -0.065892668
#> 164 5.886423 6.755703e-03 1 1.4606348198 -0.460634820 -1 -0.404385139
#> 225 3.843428 6.934491e-03 1 0.1568411180 0.843158882 1 1.420818842
#> 238 3.813541 6.991735e-03 1 0.1529636414 0.847036359 1 1.435631335
#> 23 5.074565 6.994508e-03 1 0.6161844112 0.383815589 1 0.448092407
#> 273 4.148143 7.097236e-03 1 0.2246060163 0.775393984 1 1.198343412
#> 212 5.420695 7.099505e-03 1 0.9163088936 0.083691106 1 0.086146906
#> 128 4.962366 7.101513e-03 1 0.5524403764 0.447559624 1 0.540092851
#> 242 6.034108 7.188278e-03 1 1.8262295890 -0.826229589 -1 -0.669292285
#> 7 5.487429 7.236927e-03 1 1.0049366567 -0.004936657 -1 -0.004928556
#> 222 5.002132 7.294616e-03 1 0.5924784089 0.407521591 1 0.481496119
#> 275 5.156012 7.337022e-03 1 0.7062158336 0.293784166 1 0.328786279
#> 248 4.744329 7.403683e-03 1 0.4521186687 0.547881331 1 0.701326259
#> 22 5.782136 7.505808e-03 1 1.4417783198 -0.441778320 -1 -0.389617823
#> 76 5.291446 7.579080e-03 1 0.8464345578 0.153565442 1 0.162215581
#> 236 4.901821 7.672389e-03 1 0.5569748299 0.443025170 1 0.533310526
#> 223 4.369700 7.844831e-03 1 0.3161760766 0.683823923 1 0.967090578
#> 219 5.642241 8.118891e-03 1 1.3331711142 -0.333171114 -1 -0.302028851
#> 255 5.695551 8.198243e-03 1 1.4274486906 -0.427448691 -1 -0.378311967
#> 12 5.389843 8.235466e-03 1 1.0228502731 -0.022850273 -1 -0.022678512
#> 266 5.270055 8.264447e-03 1 0.8991382152 0.100861785 1 0.104467490
#> 53 5.341120 8.365532e-03 1 0.9841099139 0.015890086 1 0.015975040
#> 175 4.665999 8.406353e-03 1 0.4690484679 0.530951532 1 0.672454669
#> 281 5.398130 8.470306e-03 1 1.0608171362 -0.060817136 -1 -0.059626200
#> 240 4.994656 8.508742e-03 1 0.6823214925 0.317678507 1 0.359376761
#> 111 4.364264 8.565672e-03 1 0.3422786960 0.657721304 1 0.910394057
#> 33 5.387672 9.010942e-03 1 1.1135420556 -0.113542056 -1 -0.109508719
#> 167 4.744973 9.601335e-03 1 0.5822856220 0.417714378 1 0.496144765
#> 67 3.195041 9.747181e-03 1 0.1066442252 0.893355775 1 1.640061711
#> 286 3.998928 9.977149e-03 1 0.2651058002 0.734894200 1 1.088790234
#> 44 5.556663 1.016483e-02 1 1.5085973482 -0.508597348 -1 -0.441400130
#> 69 5.153253 1.027810e-02 1 0.9765986891 0.023401311 1 0.023586384
#> 256 6.146228 1.039499e-02 1 2.9569695086 -1.956969509 -1 -1.321215031
#> 103 4.089078 1.059418e-02 1 0.3104306294 0.689569371 1 0.980025967
#> 153 4.681932 1.081129e-02 1 0.6094358206 0.390564179 1 0.457509464
#> 176 4.965274 1.095095e-02 1 0.8438544164 0.156145584 1 0.165104258
#> 259 5.374724 1.102584e-02 1 1.3352660852 -0.335266085 -1 -0.303761414
#> 229 5.197193 1.159346e-02 1 1.1522921992 -0.152292199 -1 -0.145182809
#> 105 4.345415 1.179439e-02 1 0.4572775006 0.542722499 1 0.692448338
#> 16 4.955066 1.179744e-02 1 0.8969302251 0.103069775 1 0.106840367
#> 235 4.940277 1.217200e-02 1 0.9095780182 0.090421982 1 0.093300820
#> 298 5.955137 1.295442e-02 1 2.9645945397 -1.964594540 -1 -1.325031521
#> 115 4.792619 1.311679e-02 1 0.8308403803 0.169159620 1 0.179766317
#> 243 5.223480 1.357681e-02 1 1.3827623041 -0.382762304 -1 -0.342575936
#> 97 3.723408 1.358351e-02 1 0.2638277854 0.736172215 1 1.092049909
#> 209 4.986600 1.363832e-02 1 1.0690892318 -0.069089232 -1 -0.067559322
#> 79 5.074341 1.371792e-02 1 1.1845666541 -0.184566654 -1 -0.174296524
#> 32 4.897212 1.390672e-02 1 0.9870684245 0.012931576 1 0.012987742
#> 276 6.130207 1.430250e-02 1 3.9599171043 -2.959917104 -1 -1.779715715
#> 231 4.014173 1.444489e-02 1 0.3861267957 0.613873204 1 0.821847033
#> 271 4.741067 1.461142e-02 1 0.8715204423 0.128479558 1 0.134435115
#> 267 4.577816 1.467212e-02 1 0.7306481594 0.269351841 1 0.298232824
#> 127 4.926436 1.486152e-02 1 1.0873256860 -0.087325686 -1 -0.084905873
#> 264 5.857410 1.489175e-02 1 3.0467432373 -2.046743237 -1 -1.365774512
#> 132 4.411106 1.510442e-02 1 0.6251339554 0.374866045 1 0.435713849
#> 88 4.695816 1.523078e-02 1 0.8631184335 0.136881567 1 0.143678782
#> 261 5.951013 1.529210e-02 1 3.4667622787 -2.466762279 -1 -1.564315301
#> 63 5.689739 1.539405e-02 1 2.6144698231 -1.614469823 -1 -1.143160960
#> 211 4.835624 1.569447e-02 1 1.0370114812 -0.037011481 -1 -0.036564476
#> 29 4.521850 1.588777e-02 1 0.7420210255 0.257978974 1 0.284248924
#> 237 5.560821 1.626491e-02 1 2.3918576234 -1.391857623 -1 -1.019595323
#> 112 4.538355 1.627771e-02 1 0.7736703743 0.226329626 1 0.246088372
#> 189 4.569788 1.630243e-02 1 0.8021860531 0.197813947 1 0.212606513
#> 100 4.526599 1.635253e-02 1 0.7670955014 0.232904499 1 0.253927052
#> 13 4.378759 1.643230e-02 1 0.6546017180 0.345398282 1 0.395803008
#> 116 4.607192 1.669169e-02 1 0.8553948326 0.144605167 1 0.152229805
#> 177 5.787640 1.715414e-02 1 3.2358444974 -2.235844497 -1 -1.457089258
#> 20 5.593901 1.736921e-02 1 2.6442183421 -1.644218342 -1 -1.159174571
#> 285 4.419386 1.760827e-02 1 0.7321589003 0.267841100 1 0.296366705
#> 279 3.683755 1.802936e-02 1 0.3324001705 0.667599830 1 0.931467523
#> 125 4.899809 1.869249e-02 1 1.3190775218 -0.319077522 -1 -0.290326976
#> 72 4.290054 1.887332e-02 1 0.6789045216 0.321095478 1 0.363811212
#> 42 4.746638 2.012913e-02 1 1.1967528133 -0.196752813 -1 -0.185153521
#> 27 4.844204 2.036010e-02 1 1.3477824545 -0.347782455 -1 -0.314075911
#> 196 3.952016 2.046926e-02 1 0.5056628339 0.494337166 1 0.612450817
#> 213 4.599753 2.063958e-02 1 1.0425473640 -0.042547364 -1 -0.041958485
#> 200 4.618924 2.065233e-02 1 1.0654989012 -0.065498901 -1 -0.064121134
#> 77 4.556708 2.141979e-02 1 1.0305937618 -0.030593762 -1 -0.030287221
#> 292 5.376370 2.146851e-02 1 2.5542576925 -1.554257693 -1 -1.110401767
#> 192 5.402012 2.179250e-02 1 2.6661241826 -1.666124183 -1 -1.170895710
#> 168 3.575505 2.181733e-02 1 0.3549146542 0.645085346 1 0.884073055
#> 137 4.699024 2.242039e-02 1 1.2606891155 -0.260689116 -1 -0.240959030
#> 296 5.134448 2.242761e-02 1 2.0400037009 -1.040003701 -1 -0.808767060
#> 293 5.244165 2.268948e-02 1 2.3289481576 -1.328948158 -1 -0.983393539
#> 58 3.955865 2.303471e-02 1 0.5694889572 0.430511043 1 0.514790916
#> 210 5.510108 2.319029e-02 1 3.1911376480 -2.191137648 -1 -1.435799544
#> 36 4.585005 2.350073e-02 1 1.1634522633 -0.163452263 -1 -0.155309941
#> 129 4.318340 2.352118e-02 1 0.8673346470 0.132665353 1 0.139032665
#> 17 4.983810 2.374099e-02 1 1.8253928431 -0.825392843 -1 -0.668726585
#> 268 3.810887 2.376964e-02 1 0.5002351298 0.499764870 1 0.621147585
#> 74 4.065902 2.434690e-02 1 0.6786243340 0.321375666 1 0.364175513
#> 244 4.121292 2.538012e-02 1 0.7511430331 0.248856967 1 0.273138139
#> 78 5.001105 2.589449e-02 1 2.0242158473 -1.024215847 -1 -0.798790908
#> 262 4.610763 2.628633e-02 1 1.3345284675 -0.334528467 -1 -0.303151599
#> 191 4.822896 2.648582e-02 1 1.6993484850 -0.699348485 -1 -0.581555761
#> 186 4.841245 2.697997e-02 1 1.7655415691 -0.765541569 -1 -0.627828142
#> 198 3.984153 2.773864e-02 1 0.7037078264 0.296292174 1 0.331963416
#> 47 5.364582 2.786027e-02 1 3.2469465426 -2.246946543 -1 -1.462348463
#> 227 4.668868 2.791929e-02 1 1.5087244839 -0.508724484 -1 -0.441497231
#> 144 4.536470 2.819286e-02 1 1.3158447263 -0.315844726 -1 -0.287631325
#> 173 5.456999 2.934729e-02 1 3.7820856320 -2.782085632 -1 -1.704001185
#> 146 4.249181 2.939380e-02 1 0.9976268257 0.002373174 1 0.002375054
#> 260 4.574732 2.990610e-02 1 1.4535504608 -0.453550461 -1 -0.398851611
#> 162 4.761353 3.042858e-02 1 1.8166015702 -0.816601570 -1 -0.662773089
#> 65 4.256727 3.117679e-02 1 1.0651582529 -0.065158253 -1 -0.063794522
#> 57 4.632779 3.160045e-02 1 1.6349661711 -0.634966171 -1 -0.535432643
#> 80 4.410479 3.230765e-02 1 1.3067564790 -0.306756479 -1 -0.280029935
#> 300 4.694185 3.260501e-02 1 1.8036821618 -0.803682162 -1 -0.653990735
#> 28 3.529248 3.288957e-02 1 0.5023007977 0.497699202 1 0.617829973
#> 118 4.720887 3.326862e-02 1 1.8943658685 -0.894365868 -1 -0.714817068
#> 195 3.791917 3.474907e-02 1 0.7082069674 0.291793033 1 0.326269424
#> 150 3.359394 3.477969e-02 1 0.4395778693 0.560422131 1 0.723212652
#> 89 5.370019 3.532239e-02 1 4.1126998898 -3.112699890 -1 -1.843160421
#> 38 4.940687 3.639618e-02 1 2.6350354645 -1.635035465 -1 -1.154243322
#> 232 2.111079 3.741333e-02 1 0.1188361379 0.881163862 1 1.580408723
#> 188 4.233918 3.822148e-02 1 1.2657694586 -0.265769459 -1 -0.245313080
#> 75 3.746367 3.847363e-02 1 0.7434157155 0.256584284 1 0.282544145
#> 158 4.378733 3.894701e-02 1 1.5127059858 -0.512705986 -1 -0.444535477
#> 10 4.694612 4.001841e-02 1 2.2015683686 -1.201568369 -1 -0.908183206
#> 253 4.930345 4.015504e-02 1 2.8658822629 -1.865882263 -1 -1.275151767
#> 140 4.943103 4.185950e-02 1 3.0262411608 -2.026241161 -1 -1.355669467
#> 46 4.292505 4.325768e-02 1 1.5227920163 -0.522792016 -1 -0.452209052
#> 98 4.426735 4.402701e-02 1 1.7966630719 -0.796663072 -1 -0.649202552
#> 54 3.870637 4.496208e-02 1 0.9920820542 0.007917946 1 0.007938941
#> 165 4.380757 4.498094e-02 1 1.7435969699 -0.743596970 -1 -0.612612057
#> 143 3.335028 4.525564e-02 1 0.5525114593 0.447488541 1 0.539986230
#> 37 4.605710 4.624488e-02 1 2.2963919398 -1.296391940 -1 -0.964419795
#> 84 3.476433 5.110284e-02 1 0.7267813307 0.273218669 1 0.303021321
#> 35 3.622940 5.139426e-02 1 0.8591823522 0.140817648 1 0.148030047
#> 184 3.532750 5.404778e-02 1 0.8166582838 0.183341716 1 0.195922498
#> 130 4.675730 5.421549e-02 1 2.8951571185 -1.895157119 -1 -1.290052508
#> 91 4.457480 6.019117e-02 1 2.5179595650 -1.517959565 -1 -1.090422568
#> 155 3.512540 6.029946e-02 1 0.8881512999 0.111848700 1 0.116313951
#> 299 4.949434 6.122222e-02 1 4.4077381018 -3.407738102 -1 -1.961823869
#> 3 4.164371 6.466188e-02 1 1.9527119033 -0.952711903 -1 -0.752984432
#> 117 4.680007 6.538887e-02 1 3.4891455753 -2.489145575 -1 -1.574476859
#> 139 4.347905 6.637455e-02 1 2.4530474343 -1.453047434 -1 -1.054245068
#> 156 3.924808 7.424849e-02 1 1.7139209989 -0.713920999 -1 -0.591839964
#> 145 3.884080 7.488925e-02 1 1.6522461131 -0.652246113 -1 -0.547924210
#> 247 4.455209 7.715725e-02 1 3.1962600322 -2.196260032 -1 -1.438247994
#> 154 4.070638 7.754922e-02 1 2.1003248852 -1.100324885 -1 -0.846442963
#> 228 3.394487 8.530838e-02 1 1.0917276689 -0.091727669 -1 -0.089064136
#> 136 3.463550 9.491171e-02 1 1.3068222788 -0.306822279 -1 -0.280085094
#> 270 4.060171 9.827059e-02 1 2.6127413304 -1.612741330 -1 -1.142227072
#> 221 3.266782 1.044329e-01 1 1.1537733893 -0.153773389 -1 -0.146530613
#> 14 2.785300 1.093166e-01 1 0.7086055396 0.291394460 1 0.325766200
#> 246 2.999835 1.137469e-01 1 0.9334093843 0.066590616 1 0.068128936
#> 190 4.073891 1.162474e-01 1 3.1224766932 -2.122476693 -1 -1.402747443
#> 121 4.494043 1.215979e-01 1 5.1883619478 -4.188361948 -1 -2.254747843
#> 149 3.713699 1.223064e-01 1 2.2035487835 -1.203548783 -1 -0.909373014
#> 141 3.085641 1.298620e-01 1 1.1670557978 -0.167055798 -1 -0.158566280
#> 291 4.609881 1.313113e-01 1 6.3533038370 -5.353303837 -1 -2.647386965
#> 134 3.481606 2.284535e-01 1 3.1273330082 -2.127333008 -1 -1.405099599
#> 205 2.714764 2.495848e-01 1 1.4607931162 -0.460793116 -1 -0.404508585
#> 277 2.710876 2.614048e-01 1 1.5213515993 -0.521351599 -1 -0.451115173
#> 24 3.010648 2.686244e-01 1 2.1753321495 -1.175332150 -1 -0.892357310
#> 294 3.573743 2.974617e-01 1 4.4735086392 -3.473508639 -1 -1.987629547
#> 257 2.596904 2.979228e-01 1 1.5229258979 -0.522925898 -1 -0.452310691
#> Hcoxsnell
#> 180 0.003333333
#> 280 0.010033520
#> 174 0.006677815
#> 203 0.023570092
#> 283 0.033844145
#> 11 0.072445389
#> 239 0.037292421
#> 272 0.020168732
#> 5 0.016778901
#> 206 0.365508216
#> 50 0.058235574
#> 178 0.247991870
#> 9 0.047709172
#> 254 0.013400523
#> 208 0.051205675
#> 70 0.394707313
#> 31 0.142844740
#> 43 0.079626741
#> 241 0.086860038
#> 109 0.061769143
#> 30 0.076029619
#> 163 0.139013323
#> 4 0.273911207
#> 110 0.309549664
#> 171 0.620941475
#> 251 0.054714447
#> 93 0.194036953
#> 21 0.170083744
#> 61 0.841764850
#> 48 0.146690894
#> 2 0.083236849
#> 90 0.097809040
#> 147 0.633402655
#> 274 1.268690846
#> 6 0.026983062
#> 126 0.174036313
#> 185 0.150551898
#> 187 0.445350937
#> 160 0.704861368
#> 15 0.327856779
#> 73 0.190004695
#> 282 0.065315242
#> 201 0.162225167
#> 216 0.549269232
#> 102 0.120073385
#> 39 0.384879244
#> 122 0.471668185
#> 55 0.520858306
#> 182 0.341810469
#> 152 0.116328067
#> 217 0.094146037
#> 41 0.112596723
#> 288 0.404632932
#> 234 0.235279853
#> 166 1.636409287
#> 95 0.774575718
#> 123 0.105175548
#> 295 0.964870214
#> 99 0.030407719
#> 199 0.214445796
#> 193 0.068873961
#> 59 0.287127237
#> 71 0.260867564
#> 142 0.572593838
#> 133 0.291571682
#> 161 0.243736551
#> 101 0.090496401
#> 169 0.135196529
#> 92 0.614768636
#> 40 0.429886746
#> 258 0.337137571
#> 60 0.154427867
#> 19 0.450559271
#> 172 0.206232209
#> 148 0.185988631
#> 252 0.296035967
#> 245 0.671745880
#> 34 0.198085536
#> 297 0.504191297
#> 159 0.040752629
#> 214 0.044224851
#> 108 0.590451403
#> 56 1.955921074
#> 94 0.487797528
#> 230 0.323248484
#> 135 0.351222285
#> 62 0.226894072
#> 183 0.318661328
#> 51 0.424784705
#> 284 0.127606369
#> 157 0.781822094
#> 85 0.498696791
#> 197 1.304838922
#> 87 0.897334409
#> 170 0.380001195
#> 68 1.464112837
#> 215 0.913795593
#> 131 0.332486409
#> 82 0.252265374
#> 96 0.560863532
#> 224 0.210330570
#> 250 0.973642144
#> 106 2.135882461
#> 289 0.532126002
#> 104 0.435014951
#> 278 0.746104164
#> 120 0.278297172
#> 290 1.075581451
#> 18 1.046733519
#> 220 0.360723527
#> 138 0.202150577
#> 265 2.255418685
#> 52 0.181988631
#> 233 0.178004567
#> 107 0.803881713
#> 113 1.887715765
#> 226 2.591850630
#> 8 0.889204328
#> 218 0.101485511
#> 194 1.056257329
#> 151 0.658800761
#> 249 0.375146826
#> 66 0.526476284
#> 181 0.123832784
#> 124 0.399657808
#> 49 0.419708563
#> 1 0.218578028
#> 204 0.300520272
#> 202 2.734924223
#> 179 1.449827123
#> 64 0.477015778
#> 86 0.639691964
#> 83 1.744619930
#> 207 1.027953762
#> 119 0.811344400
#> 269 0.355961622
#> 287 0.231078173
#> 26 0.482392122
#> 81 0.282702458
#> 263 0.409632932
#> 45 0.239499263
#> 114 0.256557220
#> 25 1.146317399
#> 164 1.508236846
#> 225 0.166146736
#> 238 0.158318917
#> 23 0.602536105
#> 273 0.222727405
#> 212 0.947553872
#> 128 0.537807820
#> 242 1.865976634
#> 7 1.037299557
#> 222 0.584463379
#> 275 0.684860775
#> 248 0.455794873
#> 22 1.478605591
#> 76 0.834072542
#> 236 0.555049579
#> 223 0.314095118
#> 219 1.368150355
#> 255 1.435742616
#> 12 1.065872713
#> 266 0.930532288
#> 53 0.991420273
#> 175 0.466349036
#> 281 1.115387373
#> 240 0.665252374
#> 111 0.370315908
#> 33 1.189104266
#> 167 0.578510998
#> 67 0.108879251
#> 286 0.269544395
#> 44 1.538772977
#> 69 0.982491702
#> 256 2.787555802
#> 103 0.305024777
#> 153 0.596475499
#> 176 0.826438954
#> 259 1.394641583
#> 229 1.200093277
#> 105 0.461058031
#> 16 0.922128926
#> 235 0.939006864
#> 298 2.843111358
#> 115 0.818863197
#> 243 1.421853727
#> 97 0.265196569
#> 209 1.156843715
#> 79 1.245298233
#> 32 1.000429282
#> 276 3.999330547
#> 231 0.414658058
#> 271 0.881139811
#> 267 0.718420846
#> 127 1.167482013
#> 264 2.964434887
#> 132 0.608633666
#> 88 0.865203303
#> 261 3.564806737
#> 63 2.428244164
#> 211 1.095286363
#> 29 0.732166712
#> 237 2.224168685
#> 112 0.767381473
#> 189 0.789121365
#> 100 0.760238616
#> 13 0.627152655
#> 116 0.849516788
#> 177 3.353695626
#> 20 2.506705703
#> 285 0.725270161
#> 279 0.346505304
#> 125 1.355163342
#> 72 0.652390504
#> 42 1.256926140
#> 27 1.408155097
#> 196 0.515271714
#> 213 1.105286363
#> 200 1.135900733
#> 77 1.085385373
#> 292 2.355492841
#> 192 2.548372369
#> 168 0.389781204
#> 137 1.280595608
#> 296 2.004120841
#> 293 2.193865655
#> 58 0.566711485
#> 210 3.179453202
#> 36 1.222440344
#> 129 0.873139811
#> 17 1.844700039
#> 268 0.493232310
#> 74 0.646021077
#> 244 0.753146417
#> 78 1.979730597
#> 262 1.381308250
#> 191 1.689051668
#> 186 1.763850699
#> 198 0.678281828
#> 47 3.453695626
#> 227 1.554397977
#> 144 1.342342829
#> 173 3.832663880
#> 146 1.018694503
#> 260 1.493311473
#> 162 1.823866705
#> 65 1.125591454
#> 57 1.653650666
#> 80 1.317184601
#> 300 1.803458542
#> 28 0.509716159
#> 118 1.909937987
#> 195 0.691483292
#> 150 0.440169590
#> 89 4.199330547
#> 38 2.466705703
#> 232 0.131394247
#> 188 1.292643800
#> 75 0.739111157
#> 158 1.570270992
#> 10 2.081077656
#> 253 2.637305176
#> 140 2.901934887
#> 46 1.602793467
#> 98 1.783458542
#> 54 1.009520191
#> 165 1.725752005
#> 143 0.543522105
#> 37 2.164453890
#> 84 0.711618124
#> 35 0.857329288
#> 184 0.796474306
#> 130 2.684924223
#> 91 2.321010083
#> 155 0.905531130
#> 299 4.449330547
#> 3 1.932665260
#> 117 3.689806737
#> 139 2.287676749
#> 156 1.707233487
#> 145 1.671194526
#> 247 3.262786535
#> 154 2.029120841
#> 228 1.178234701
#> 136 1.329684601
#> 270 2.391207127
#> 221 1.211204389
#> 14 0.698149958
#> 246 0.956174562
#> 190 3.031101554
#> 121 5.282663880
#> 149 2.108104684
#> 141 1.233803980
#> 291 6.282663880
#> 134 3.102530125
#> 205 1.523388361
#> 277 1.586400025
#> 24 2.054761867
#> 294 4.782663880
#> 257 1.619460134timeroc_predictFinally, after fitting process, we can predict the value of
sensitivity and specificity of the covariates at specific time point
using the parTimeROC::timeroc_predict() function. This will
return a list of dataframe for each specified time.
To generate the ROC curve, user can choose to conduct the prediction
procedure using the type = 'standard' or
type = 'landmark' approach.
By default, the type = 'standard' analysis will be used
to produce the ROC curve at different time point. After model fitting
procedure, the estimated parameters will be extracted and used to
compute the ROC at the specified time of interest.
Meanwhile for the type = 'landmark' analysis, at each
time point of interest, the status of each observation will be updated
prior running the model fitting procedure. Hence, in landmark analysis,
the fitting procedure will be conducted multiple times. At each time of
interest, the updated estimators are then used to produce the ROC
curve.
library(parTimeROC)
# Copula model
test <- timeroc_obj(dist = 'gompertz-gompertz-copula', copula='clayton90',
params.t = c(shape=3,rate=1),
params.x = c(shape=1,rate=2),
params.copula=-5)
set.seed(23456)
rr <- rtimeroc(obj = test, censor.rate = 0.2, n=500)
cc <- timeroc_fit(x=rr$x, t=rr$t, event=rr$event, obj = test)
jj <- timeroc_predict(cc, t = quantile(rr$t,probs = c(0.25, 0.5)))
plot(x = 1-jj[[1]][,2], y = jj[[1]][,1], type = 'l')
lines(x = 1-jj[[2]][,2], y = jj[[2]][,1], col = 'blue')
Fig.6. ROC curve at 25th & 50th quantile points of time-to-event
We can also specify the number of bootstrap process that we want if
confidence interval of the ROC curve need to be computed. The bootstrap
procedure can be achieved by supplying B = bootstrap value
into the parTimeROC::timeroc_predict() function.
library(parTimeROC)
# Copula model
test <- timeroc_obj(dist = 'gompertz-gompertz-copula', copula='clayton90',
params.t = c(shape=3,rate=1),
params.x = c(shape=1,rate=2),
params.copula=-5)
set.seed(23456)
rr <- rtimeroc(obj = test, censor.rate = 0.2, n=500)
cc <- timeroc_fit(x=rr$x, t=rr$t, event=rr$event, obj = test)
jj <- timeroc_predict(cc, t = quantile(rr$t,probs = c(0.25)), B = 500)
plot(x = 1-jj[[1]][,2], y = jj[[1]][,1], type = 'l')
lines(x = 1-jj[[1]][,4], y = jj[[1]][,3], col = 'red')
lines(x = 1-jj[[1]][,6], y = jj[[1]][,5], col = 'red')
Fig.7. 95% boot confidence interval of ROC curve at 25th time-to-event
timeroc_aucFunction to compute the area under the ROC curve using the
parTimeROC::timeroc_auc() is also prepared for user
convenience.
test <- timeroc_obj('normal-weibull-copula', copula = 'clayton90')
print(test)
#> Model Assumptions: 90 Degrees Rotated Clayton Copula
#> X : Gaussian
#> Time-to-Event : Weibull
set.seed(23456)
rr <- rtimeroc(obj = test, censor.rate = 0.1, n=500,
params.t = c(shape=1, scale=5),
params.x = c(mean=5, sd=1),
params.copula=-2)
cc <- timeroc_fit(x=rr$x, t=rr$t, event=rr$event, obj = test)
jj <- timeroc_predict(cc, t = quantile(rr$t, probs = c(0.25,0.5,0.75)),
B = 500)
print(timeroc_auc(jj))
#> time assoc est.auc low.auc upp.auc
#> 1 1.671625 -1.889754 0.8871412 0.8360745 0.9251444
#> 2 3.822324 -1.889754 0.8204138 0.7650090 0.8657244
#> 3 7.396509 -1.889754 0.7725274 0.7156493 0.8204064