Derivation and validation of pragmatic clinical models to predict hospital length of stay after cardiac surgery in Ontario, Canada: a population-based cohort study =================================================================================================================================================================== * Alexandra Fottinger * Anan Bader Eddeen * Douglas S. Lee * Graham Woodward * Louise Y. Sun ## Abstract **Background:** Cardiac surgery is resource intensive and often requires multidisciplinary involvement to facilitate discharge. To facilitate evidence-based resource planning, we derived and validated clinical models to predict postoperative hospital length of stay (LOS). **Methods:** We used linked, population-level databases with information on all Ontario residents and included patients aged 18 years or older who underwent coronary artery bypass grafting, valvular or thoracic aorta surgeries between October 2008 and September 2019. The primary outcome was hospital LOS. The models were derived by using patients who had surgery before Sept. 30, 2016, and validated after that date. To address the rightward skew in LOS data and to identify top-tier resource users, we used logistic regression to derive a model to predict the likelihood of LOS being more than the 98th percentile (> 30 d), and γ regression in the remainder to predict continuous LOS in days. We used backward stepwise variable selection for both models. **Results:** Among 105 193 patients, 2422 (2.3%) had an LOS of more than 30 days. Factors predicting prolonged LOS included age, female sex, procedure type and urgency, comorbidities including frailty, high-risk acute coronary syndrome, heart failure, reduced left ventricular ejection fraction and psychiatric and pulmonary circulatory disease. The C statistic was 0.92 for the prolonged LOS model and the mean absolute error was 2.4 days for the continuous LOS model. **Interpretation:** We derived and validated clinical models to identify top-tier resource users and predict continuous LOS with excellent accuracy. Our models could be used to benchmark clinical performance based on expected LOS, rationally allocate resources and support patient-centred operative decision-making. Cardiac surgery is resource intensive. It carries a higher burden of complications, requires intensive postoperative monitoring and involves an often longer hospital length of stay (LOS) as compared with non-cardiac surgery.1 With steady improvements in surgical technique and perioperative care, cardiac surgery is increasingly being offered to frail and complex patients with higher resource needs.2,3 The drive by many organizations for operational efficiency and competing capacity needs during the COVID-19 pandemic makes evidence-based triaging and resource allocation, founded on real-world data, an urgent priority. Prediction of intensive care unit (ICU) LOS after cardiac surgery4–6 is important but does not fully reflect the extent of resources needed. Nonetheless, although risk factors have been identified for prolonged postoperative hospital LOS, few models are available to predict this important metric, and none are able to estimate continuous postoperative LOS with accuracy. Further, though existing models include those from the Society of Thoracic Surgeons and the EuroSCORE data sets,7–9 they were developed to predict perioperative death and end organ morbidity and were only later validated in single-centre data sets for the purpose of predicting prolonged LOS in a binary fashion instead of estimation of continuous LOS. To better inform health resource planning, we derived and externally validated clinical models using population-based data to identify top-tier resource users and to predict continuous hospital LOS after cardiac surgery. ## Methods We conducted a population-based, retrospective cohort study in Ontario between Oct. 1, 2008, and Sept. 30, 2019. Ontario is the most populous province in Canada, with about 14.6 million residents, and is ethnically diverse.10 Ontario operates a publicly funded health care system that reimburses all covered services and clinicians. All centres in Ontario performing cardiac surgical procedures were included. The conduct and reporting of this study is in accordance with TRIPOD (Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis) Statement11 reporting guideline. ### Participants The population included all adult patients aged 18 years or older who underwent coronary artery bypass grafting (CABG), aortic, mitral or tricuspid valve, or thoracic aorta surgery. These procedures represent the majority of cardiac surgeries performed in the United States and are frequently the target for risk prediction models in the cardiac surgical literature.12 For patients with multiple cardiac procedures during the study period, only the index procedure was included in the analyses. ### Data source The data set from this study is held securely in coded form at ICES (formerly the Institute for Clinical Evaluative Sciences), which holds multiple population-based health databases of Ontario residents. CorHealth Ontario maintains a prospective registry of all patients who undergo invasive cardiac procedures in Ontario and regularly undergoes selected chart audits and core laboratory validation.13 Data sets were linked deterministically using unique confidential identifiers and were analyzed at ICES.14–17 These included the CorHealth Ontario registry (patient and procedural details) with population-level administrative health care databases including the Canadian Institute for Health Information Discharge Abstract Database (comorbidities, hospital admissions and in-hospital procedures), the Ontario Health Insurance Plan database (physician service claims) and the Registered Persons Database (vital statistics).17–20 These administrative databases have been validated for many outcomes, exposures and comorbidities, including heart failure, chronic obstructive pulmonary disease, asthma, hypertension and diabetes.21–24 ### Covariates Potential covariates considered in the analyses included patient sociodemographics, comorbidities and hospital characteristics. We estimated each patient’s socioeconomic status by using the neighbourhood median income from the Canadian Census,25 and determined residence status (rural v. urban) using Statistics Canada definitions.26 The type of hospital (teaching and community) was determined by the *Public Hospitals Act*, where teaching hospitals had formal agreements with a university and postgraduate training accreditation body.27 Further variables included physiologic, anatomic and comorbidity data, as well as procedure-specific information (operative priority status, redo sternotomy, type of surgery and surgery duration). We obtained data on height, weight, operative priority and information pertaining to left ventricular ejection fraction, valvular disease and coronary anatomy from the CorHealth Ontario registry. In addition, we identified comorbidities from the CorHealth Ontario registry, which we supplemented with data from the Discharge Abstract Database and the Ontario Health Insurance Plan database using the enhanced Canadian version of the *International Classification of Diseases and Related Health Problems, 10th Revision* codes28 within 5 years before surgery, according to validated algorithms.21,23,29,30 Frailty is defined by the Hospital Frailty Risk Score, a multidimensional frailty instrument that was validated using routinely collected hospital admissions data of patients aged 75 years or older from the United Kingdom.31,32 This score ranges between 0 and 173.1 points and is stratified by low risk (< 5), intermediate risk3–13 and high risk (> 15).31 ### Outcome The primary outcome was postoperative hospital LOS, from the date of surgery to the date of index hospital discharge. We derived 2 separate models: the binary outcome model to identify the top-tier resource users (i.e., LOS exceeding the 98th percentile value of > 30 d) and the continuous outcome model to predict the actual LOS in days in the remainder of the cohort. The rationale for our modelling approach was twofold. First, as LOS data are invariably right-skewed with extreme values in patients with prolonged stay,33,34 the choice of 98th percentile (LOS > 30 d) as the cut-off point eliminates extreme values and increases the precision of continuous LOS prediction. Second, patients with LOS of more than 30 days are among the highest resource users and are most likely to have impaired quality of life after eventual discharge.13,35–38 Thus, the ability to predict prolonged LOS will allow patients to be informed partners in the therapeutic decision-making process. ### Statistical analysis We compared continuous variables using a 2-sample *t* test or Wilcoxon rank-sum test where appropriate, and categorical variables using a χ2 test. Between-group differences were considered to be small if the absolute standardized difference for a given variable was less than 10%.39 #### Missing data The left ventricular ejection fraction was missing in 3582 (3.4%), rurality status in 87 (0.08%), income quintile in 272 (0.26%), glomerular filtration rate in 4671 (4.4%), body mass index (BMI) in 5583 (5.3%), surgery duration in 1317 (1.2%) and operative priority in 12 060 (11.5%) patients. We imputed these missing values once within the SAS PROC MI framework, where they were predicted drawing on all candidate covariates using predictive mean matching for continuous variables and logistic regression for categorical variables.40 #### Model development and validation We split the cohort temporally into derivation and validation data sets, such that the cohort who underwent cardiac surgery before Sept. 30, 2016, was used to derive the models and the remainder of the cohort was used to externally validate these models. We predicted prolonged hospital LOS using logistic regression and continuous hospital LOS using γ regression. For each model, we selected predictor variables using a backward-stepwise algorithm with a significance threshold (*p* < 0.10 for entry and *p* < 0.05 for retention in the model).41 For continuous variables, we examined their association with the outcome using cubic splines. Most of these variables were entered into the models as continuous values, whereas BMI violated the linearity assumption and was entered as a spline term. #### Model evaluation The discrimination of the binary outcome model was evaluated using the C statistic, and its discrimination and calibration was assessed using the Brier score,42 as well as plots of observed compared with predicted rates within vigintiles of predicted risk in the validation cohort. The performance of the continuous outcome model was assessed using the mean absolute error (MAE), as well as plots of mean observed compared with predicted LOS in days within each vigintile of predicted LOS in patients from the validation cohort with high likelihood of LOS of more than 30 days (i.e., probability exceeding the optimal cut-off value as determined by the Youden Index). We performed the analysis using SAS version 9.4 (SAS Institute) and defined statistical significance by a 2-sided *p* value of less than 0.05. ### Ethics approval An independent, nonprofit research institute, ICES holds legal status under Ontario’s health information privacy law, which allows it to collect and analyze health care and demographic data, without consent, for health system evaluation and improvement. The use of data was authorized under section 45 of Ontario’s *Personal Health Information Protection Act*, which does not require review by a research ethics board.13 ## Results Among 105 193 patients (73 870 comprising the derivation cohort and 28 901 comprising the validation cohort), 2422 (2.3%) had prolonged hospital LOS of more than 30 days (Table 1). The distribution of hospital LOS in the derivation and validation cohorts are illustrated in Appendix 1, available at [www.cmajopen.ca/content/11/1/E180/suppl/DC1](http://www.cmajopen.ca/content/11/1/E180/suppl/DC1). The median LOS was 46 (interquartile range [IQR] 37–66) days for patients with prolonged LOS and 6 (IQR 5–8) days for the remainder of the cohort. Patient characteristics were notably different between groups (Table 1). A minority of the patients in either group were female, though females were more likely than males to have prolonged hospital LOS. Patients with prolonged hospital LOS were more likely to be older, with a 6-year median age difference noted, and reside in a lower income neighbourhood, and were more likely to present urgently and emergently for complex procedures (re-do sternotomy, CABG + valve[s], multiple valves and thoracic aorta surgery) of longer surgical duration and at teaching hospitals. They were more likely to have a higher multimorbidity burden, including frailty, in addition to cardiovascular comorbidities such as recent myocardial infarction, reduced left ventricular ejection fraction and higher New York Heart Association classification. In addition, prolonged hospital LOS was associated with longer mechanical ventilation and ICU LOS, as well as higher rates of surgical re-exploration, extracorporeal membrane oxygenation, surgical site infection, ventilator-associated pneumonia, acute kidney injury, myocardial infarction, stroke, and physician visits and readmission to ICU and hospital within 30 days of discharge (Appendix 2, available at [www.cmajopen.ca/content/11/1/E180/suppl/DC1](http://www.cmajopen.ca/content/11/1/E180/suppl/DC1)). View this table: [Table 1:](http://www.cmajopen.ca/content/11/1/E180/T1) Table 1: Baseline characteristics of patients by hospital length of stay ### Predictors of length of stay The binary model for prolonged LOS consisted of 16 variables (Table 2) with the continuous model consisting of 28 variables (Table 3). The characteristics common to both models are procedure type and duration, age, rural residence, BMI, frailty, Canadian Cardiovascular Society and New York Heart Association classification status, left ventricular ejection fraction, glomerular filtration rate, valvular disease, diabetes requiring treatment, anemia, cerebrovascular disease, malignancy and depression. The continuous outcome model additionally included sex, presenting at a community hospital, operative priority, atrial fibrillation, endocarditis, peripheral arterial disease, chronic obstructive pulmonary disease, pulmonary circulatory disease, alcoholism, dementia and psychosis. View this table: [Table 2:](http://www.cmajopen.ca/content/11/1/E180/T2) Table 2: Multivariable predictors of prolonged hospital length of stay of more than 30 days View this table: [Table 3:](http://www.cmajopen.ca/content/11/1/E180/T3) Table 3: Multivariable predictors of a continuous model describing continuous hospital length of stay in days ### Model performance #### Binary outcome model The C statistic was 0.92 in both derivation and validation data sets, demonstrating excellent discrimination. The model was well calibrated, with a Brier score of 0.016 and the observed and predicted risks of prolonged LOS being very similar across all probability vigintiles in the calibration data set (Figure 1). At the optimal cut-off value of 0.0166, the binary model had a sensitivity of 88.7% and specificity of 81.2%. ![Figure 1:](http://www.cmajopen.ca/https://www.cmajopen.ca/content/cmajo/11/1/E180/F1.medium.gif) [Figure 1:](http://www.cmajopen.ca/content/11/1/E180/F1) Figure 1: Calibration plot of observed compared with predicted risk of extremely prolonged postoperative hospital LOS of more than 30 days, according to vigintiles of expected rate in (A) the derivation cohort and (B) validation cohort. Note: LOS = length of stay. #### Continuous outcome model The continuous model had an MAE of 2.3 days in the derivation data set. The MAE was 2.0 in the validation data set, indicating good predictive accuracy. The distribution of MAE according to vigintiles of hospital LOS in the derivation and validation cohorts are presented in Appendix 3 (available at [www.cmajopen.ca/content/11/1/E180/suppl/DC1](http://www.cmajopen.ca/content/11/1/E180/suppl/DC1)). The calibration plot in Figure 2 shows that the mean observed and predicted hospital LOS within each LOS vigintile were nearly identical in the validation cohort. ![Figure 2:](http://www.cmajopen.ca/https://www.cmajopen.ca/content/cmajo/11/1/E180/F2.medium.gif) [Figure 2:](http://www.cmajopen.ca/content/11/1/E180/F2) Figure 2: Calibration plot of observed compared with predicted average of hospital LOS in days, within each vigintile of predicted length of stay in (A) the derivation cohort and (B) validation cohort. Note: LOS = length of stay. ## Interpretation Operative decision-making, performance benchmarking and postoperative resource planning may be enhanced by objective tools. Traditional statistical models are dated; they are limited to the prediction of prolonged LOS of varying durations and fail to predict continuous LOS.43–46 Although continuous hospital LOS is beginning to be explored using machine learning techniques in isolated CABG patients, it is based on small, single-centre data sets and lacks generalizability in the broader health care setting.47,48 The ability to predict continuous hospital LOS is advantageous to hospital administrators, patients and caregivers. Patients often ask about their projected LOS as it sets realistic expectations of the costs and benefits of surgery as a part of the informed consent and therapeutic decision-making process. For caregivers, an estimated LOS will more effectively facilitate personal and professional arrangements to care for loved ones after surgery. At the hospital level, the widening use of electronic medical records may lend itself to institution-based, self-quality assessments, and to rationally allocate telemonitoring and other postdischarge care and follow-up resources to reduce morbidity and readmissions. At the system level, these models could be used to generate risk-adjusted observed compared with expected LOS to systematically evaluate performance. Our models were pragmatically designed for operational capacity planning and were derived and validated in a large and representative population to overcome these limitations. CorHealth Ontario produced an independent report addressing risk-adjusted average hospital LOS for the 2011–2016 fiscal years.49 Of note, this report had trimmed hospital LOS at the 99th percentile to remove extreme observations, much like the methodology employed in our analysis to isolate toptier resource users who are at the highest risk of complications, worsening frailty, functional decline and loss of personal freedom and independence after surgery.13,36,37,50–52 The ability to identify patients at risk for prolonged LOS allows for better decision-making from the health care system as well as the patient. At the system level, this ability, coupled with actual LOS prediction, will facilitate data-driven clinical scheduling to increase throughput, facilitate targeted interventions such as prehabilitation, Enhanced Recovery After Surgery and early referral to continuing care facilities. As prolonged LOS has also been implicated with increased health care cost7 and disability after discharge,13,35–38 our predictive models will inform effective provider–patient discussions and encourage patient-centred operative decision-making. Practically, this could be implemented by first identifying patients at risk of prolonged LOS followed by using the continuous model to facilitate actual LOS prediction in patients whose LOS would not likely exceed 30 days. Notably, our binary outcome model demonstrated excellent performance with a C statistic of 0.92 and outperforms existing models. Comparatively, the EuroScore had a C statistic of 0.71 (IQR 0.69–0.72) for predicting prolonged hospital LOS (> 12 d) when validated in a monocentric setting,9 and the Society of Thoracic Surgeon’s model had a C statistic of 0.716–0.732 for predicting a short LOS of 5 days or less and 0.739–0.796 for predicting a prolonged LOS of more than 5 days, depending on the type of surgery performed.53 It should also be noted that these models rely on designated staff for data collection, which constitutes further health care resource demands and is not feasible at all Canadian centres. Our continuous outcome model was able to predict LOS with an MAE of 2 days, which is accepted in a publicly funded health care system given LOS could be influenced by the availability of postdischarge continuing care facilities and home-based caregivers rather than medical indications alone. Importantly, our ability to predict continuous LOS enables precision-based hospital capacity planning, as well as quality benchmarking and incentivized allocation of health care funding. Incorporation of the models into tools, such as the province-wide CorHealth information system, could also help individual providers to understand bed requirements at the time of intervention, allowing for more accurate resource planning. Many risk factors from our LOS models are consistent with those published in the literature.11,54 In addition, we were able to incorporate frailty as a defining element of perioperative outcomes and recovery,55,56 as well as anemia, hospital type and a variety of sociodemographic factors to ensure that all patient groups are equally represented. A further novel component was the inclusion of psychiatric comorbidities such as depression and psychosis, which have been demonstrated in other settings to be associated with increased hospital admission, LOS and death.57,58 The apparent protective influence of rurality on outcomes is likely explained by observations from other Ontario population-based studies, that rural residents are more likely to be higher income earners, younger and less likely to be frail or have comorbidities such as anemia and renal disease.59 The variables included in our models are routinely collected and readily available to facilitate their adoption at most institutions. Notable strengths of our models include their generalizability across the scope of the most frequently performed cardiac surgical procedures in a large and representative population, as well as their relevance to clinicians, administrators, policy-makers and patients. ### Limitations Limitations include those inherent to health administrative data sets including the lack of certain detailed physiologic measures, such as natriuretic peptide, in the databases used. Additional limitations include limited application in patients who undergo minimally invasive cardiac procedures and less frequently performed procedures such as cardiac transplant and ventricular assist device placement. Further validation may be needed in other health care jurisdictions. ### Conclusion We derived and validated a set of clinical prediction models to identify top-tier resource users (hospital LOS > 30 d) and actual LOS after cardiac surgery with excellent accuracy. Care, outcomes and patient satisfaction may be substantially improved if clinical judgment is supported by objective quantification in the planning of care. Shared decision-making through a thorough discussion of evidence-based estimates of risk and benefits of treatment options with patients and caregivers is vital to patient-centred care. Our research provides scientific evidence to inform shared clinical decision-making. Also, being based on an unbiased population-based sample, these models could be combined with established ICU LOS7 and waitlist60 management tools to provide evidence-based triaging decision support, to conserve system capacity, enhance operational efficiency, and to benchmark performance. ## Acknowledgement The authors acknowledge the usage of data compiled and provided by the Canadian Institute for Health Information. These data sets were linked using unique encoded identifiers and analyzed at ICES. The analyses, conclusions, opinions and statements expressed in the manuscript are those of the authors and do not necessarily reflect those of the above agencies. ## Footnotes * **Competing interests:** None declared. * This article has been peer reviewed. * **Contributors:** Louise Sun contributed to the conception and design of the work. Anan Bader Eddeen and Louise Sun contributed to the acquisition and analysis of the data. Alexandra Fottinger, Anan Bader Eddeen and Louise Sun interpreted the data. Alexandra Fottinger and Louise Sun drafted the manuscript. All authors revised the manuscript critically for important intellectual content, gave final approval of the version to be published and agreed to be accountable for all aspects of the work. * **Funding:** This study is supported by the COVID-19 Response Fund from the University of Ottawa. Louise Sun was the National New Investigator at the Heart and Stroke Foundation of Canada. Douglas Lee is the Ted Rogers Chair in Heart Function Outcomes, University Health Network, University of Toronto. * **Data sharing:** The data set from this study is held securely in coded form at ICES. Although legal data sharing agreements between ICES and data providers (e.g., health care organizations and government) prohibit ICES from making the data set publicly available, access may be granted to those who meet prespecified criteria for confidential access, available at [https://www.ices.on.ca/DAS](https://www.ices.on.ca/DAS) (email: das{at}ices.on.ca). The full data set creation plan and underlying analytic code are available from the authors on request, understanding that the computer programs may rely on coding templates or macros that are unique to ICES and are therefore either inaccessible or may require modification. * **Disclaimer:** This study was supported by ICES, which is funded by an annual grant from the Ontario Ministry of Health (MOH) and the Ministry of Long-Term Care (MLTC). Parts of this material are based on data and information compiled and provided by the Canadian Institute for Health Information (CIHI). The authors acknowledge that the clinical registry data used in this analysis is from participating hospitals through CorHealth Ontario, which serves as an advisory body to the MOH, is funded by the MOH and is dedicated to improving the quality, efficiency and access and equity in the delivery of the continuum of adult cardiac and stroke care in Ontario. The analyses, conclusions, opinions and statements expressed herein are solely those of the authors and do not reflect those of the funding or data sources; no endorsement is intended or should be inferred. * **Supplemental information:** For reviewer comments and the original submission of this manuscript, please see [www.cmajopen.ca/content/11/1/E180/suppl/DC1](http://www.cmajopen.ca/content/11/1/E180/suppl/DC1). This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY-NC-ND 4.0) licence, which permits use, distribution and reproduction in any medium, provided that the original publication is properly cited, the use is noncommercial (i.e., research or educational use) and no modifications or adaptations are made. See: [https://creativecommons.org/licenses/by-nc-nd/4.0/](https://creativecommons.org/licenses/by-nc-nd/4.0/) ## References 1. Fry DE, Pine M, Jones BL, et al. (2009) Adverse outcomes in surgery: redefinition of postoperative complications. Am J Surg 197:479–84. [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1016/j.amjsurg.2008.07.056&link_type=DOI) [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=19246026&link_type=MED&atom=%2Fcmajo%2F11%2F1%2FE180.atom) [Web of Science](http://www.cmajopen.ca/lookup/external-ref?access_num=000264999600010&link_type=ISI) 2. Montgomery C, Stelfox H, Norris C, et al. (2021) Association between preoperative frailty and outcomes among adults undergoing cardiac surgery: a prospective cohort study. CMAJ Open 9:E777–87. [Abstract/FREE Full Text](http://www.cmajopen.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NToiY21ham8iO3M6NToicmVzaWQiO3M6ODoiOS8zL0U3NzciO3M6NDoiYXRvbSI7czoyMToiL2NtYWpvLzExLzEvRTE4MC5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 3. Buth KJ, Gainer RA, Legare J-F, et al. (2014) The changing face of cardiac surgery: practice patterns and outcomes 2001–2010. Can J Cardiol 30:224–30. [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1016/j.cjca.2013.10.020&link_type=DOI) [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=24373760&link_type=MED&atom=%2Fcmajo%2F11%2F1%2FE180.atom) 4. Rotar EP, Beller JP, Smolkin ME, et al. (2022) Prediction of prolonged intensive care unit length of stay following cardiac surgery. Semin Thorac Cardiovasc Surg 34:172–9. 5. Sun LY, Bader Eddeen A, Ruel M, et al. (2020) Derivation and validation of a clinical model to predict intensive care unit length of stay after cardiac surgery. J Am Heart Assoc 9:e017847. 6. Meadows K, Gibbens R, Gerrard C, et al. (2018) Prediction of patient length of stay on the intensive care unit following cardiac surgery: a logistic regression analysis based on the cardiac operative mortality risk calculator, Euro-SCORE. J Cardiothorac Vasc Anesth 32:2676–82. 7. Osnabrugge RL, Speir AM, Head SJ, et al. (2014) Prediction of costs and length of stay in coronary artery bypass grafting. Ann Thorac Surg 98:1286–93. [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1016/j.athoracsur.2014.05.073&link_type=DOI) [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=25110338&link_type=MED&atom=%2Fcmajo%2F11%2F1%2FE180.atom) 8. Peterson ED, Coombs LP, Ferguson TB, et al. (2002) Hospital variability in length of stay after coronary artery bypass surgery: results from the Society of Thoracic Surgeon’s National Cardiac Database. Ann Thorac Surg 74:464–73. [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1016/S0003-4975(02)03694-9&link_type=DOI) [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=12173830&link_type=MED&atom=%2Fcmajo%2F11%2F1%2FE180.atom) [Web of Science](http://www.cmajopen.ca/lookup/external-ref?access_num=000177320600042&link_type=ISI) 9. Toumpoulis IK, Anagnostopoulos CE, Swistel DG, et al. (2005) Does EuroSCORE predict length of stay and specific postoperative complications after cardiac surgery? Eur J Cardiothorac Surg 27:128–33. [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1016/j.ejcts.2004.09.020&link_type=DOI) [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=15621484&link_type=MED&atom=%2Fcmajo%2F11%2F1%2FE180.atom) 10. Sun LY, Boet S, Chan V, et al. (2021) Impact of surgeon and anaesthesiologist sex on patient outcomes after cardiac surgery: a population-based study. BMJ Open 11:e051192. [Abstract/FREE Full Text](http://www.cmajopen.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NzoiYm1qb3BlbiI7czo1OiJyZXNpZCI7czoxMjoiMTEvOC9lMDUxMTkyIjtzOjQ6ImF0b20iO3M6MjE6Ii9jbWFqby8xMS8xL0UxODAuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 11. Collins GS, Reitsma JB, Altman DG, et al. (2015) Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): the TRIPOD statement. Ann Intern Med 162:55–63. [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.7326/M14-0697&link_type=DOI) [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=25560714&link_type=MED&atom=%2Fcmajo%2F11%2F1%2FE180.atom) 12. D’Agostino RS, Jacobs JP, Badhwar V, et al. (2018) The Society of Thoracic Surgeons Adult Cardiac Surgery database: 2018 update on outcomes and quality. Ann Thorac Surg 105:15–23. [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1016/j.athoracsur.2017.10.035&link_type=DOI) [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=29233331&link_type=MED&atom=%2Fcmajo%2F11%2F1%2FE180.atom) 13. Sun LY, Spence SD, Benton S, et al. (2022) Age, not sex, modifies the effect of frailty on long-term outcomes after cardiac surgery. Ann Surg 275:800–6. 14. Tu JV, Chu A, Donovan LR, et al. (2015) The Cardiovascular Health in Ambulatory Care Research Team (CANHEART): using big data to measure and improve cardiovascular health and healthcare services. Circ Cardiovasc Qual Outcomes 8:204–12. [Abstract/FREE Full Text](http://www.cmajopen.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6ODoiY2lyY2N2b3EiO3M6NToicmVzaWQiO3M6NzoiOC8yLzIwNCI7czo0OiJhdG9tIjtzOjIxOiIvY21ham8vMTEvMS9FMTgwLmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 15. Harron K, Dibben C, Boyd J, et al. (2017) Challenges in administrative data linkage for research. Big Data Soc 4:2053951717745678. [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fcmajo%2F11%2F1%2FE180.atom) 16. Iron K, Zagorski B, Sykora K (2008) Living and dying in Ontario: an opportunity to improve health information (ICES, Toronto). 17. Schull MJ, Azimaee M, Marra M, et al. (2020) ICES: data, discovery, better health. Int J Popul Data Sci 4:1135. 18. Chiu M, Lebenbaum M, Lam K, et al. (2016) Describing the linkages of the immigration, refugees and citizenship Canada permanent resident data and vital statistics death registry to Ontario’s administrative health database. BMC Med Inform Decis Mak 16:135. [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1186/s12911-016-0375-3&link_type=DOI) [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=27769227&link_type=MED&atom=%2Fcmajo%2F11%2F1%2FE180.atom) 19. Fang J, Qiu F, Wang X, et al. (2018) Int J Popul Data Sci, Internal and external data linkage of complex relational database: results from CorHealth Ontario. Available: [https://ijpds.org/article/view/710](https://ijpds.org/article/view/710), p 3. accessed 2022 Apr. 9. 20. (2021) Discharge Abstract Database, 2019–2020 and 2020–2021 Abacus Data Network, V1, UNF:6:g/k+/5S9AnzFOhXd58GNig==[fileUNF] (Canadian Institute for Health Information, Ottawa) Available: [https://hdl.handle.net/11272.1/AB2/RQKUYZ](https://hdl.handle.net/11272.1/AB2/RQKUYZ). accessed 2022 Apr. 9. 21. Tu K, Campbell NR, Chen Z-L, et al. (2007) Accuracy of administrative databases in identifying patients with hypertension. Open Med 1:e18–26. [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=20101286&link_type=MED&atom=%2Fcmajo%2F11%2F1%2FE180.atom) 22. Juurlink D, Preya C, Croxford R, et al. (2006) Canadian Institute for Health Information Discharge Abstract Database: a validation study (ICES, Toronto). 23. Hux JE, Ivis F, Flintoft V, et al. (2002) Diabetes in Ontario: determination of prevalence and incidence using a validated administrative data algorithm. Diabetes Care 25:512–6. [Abstract/FREE Full Text](http://www.cmajopen.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NzoiZGlhY2FyZSI7czo1OiJyZXNpZCI7czo4OiIyNS8zLzUxMiI7czo0OiJhdG9tIjtzOjIxOiIvY21ham8vMTEvMS9FMTgwLmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 24. Austin PC, Daly PA, Tu JV (2002) A multicenter study of the coding accuracy of hospital discharge administrative data for patients admitted to cardiac care units in Ontario. Am Heart J 144:290–6. [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1067/mhj.2002.123839&link_type=DOI) [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=12177647&link_type=MED&atom=%2Fcmajo%2F11%2F1%2FE180.atom) [Web of Science](http://www.cmajopen.ca/lookup/external-ref?access_num=000177501400017&link_type=ISI) 25. (2012) Health Analytics Branch. Health Analyst’s Toolkit (Ontario Ministry of Health and Long Term Care, Toronto) Available: [https://www.health.gov.on.ca/english/providers/pub/healthanalytics/health\_toolkit/health_toolkit.pdf](https://www.health.gov.on.ca/english/providers/pub/healthanalytics/health_toolkit/health_toolkit.pdf). accessed 2015 Nov 14. 26. du Plessis V, Beshiri R, Bollman RD, et al. (2002) Definitions of “rural”. Cat no. 21-601-MIE: no 061 (Statistics Canada, Ottawa) Available[https://ageconsearch.umn.edu/bitstream/28031/1/wp020061.pdf](https://ageconsearch.umn.edu/bitstream/28031/1/wp020061.pdf). accessed 2022 Apr. 11. 27. Reg O Public Hospitals Act, R.S.O. 1990. c. 40, 321/01: classification of hospitals. Available: [https://www.ontario.ca/laws/regulation/r01321](https://www.ontario.ca/laws/regulation/r01321). accessed 2022 Apr. 11. 28. Quan H, Sundararajan V, Halfon P, et al. (2005) Coding algorithms for defining comorbidities in ICD-9-CM and ICD-10 administrative data. Med Care 43:1130–9. [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1097/01.mlr.0000182534.19832.83&link_type=DOI) [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=16224307&link_type=MED&atom=%2Fcmajo%2F11%2F1%2FE180.atom) [Web of Science](http://www.cmajopen.ca/lookup/external-ref?access_num=000233268500010&link_type=ISI) 29. Gershon AS, Wang C, Guan J, et al. (2009) Identifying individuals with physcian diagnosed COPD in health administrative databases. COPD 6:388–94. [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1080/15412550903140865&link_type=DOI) [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=19863368&link_type=MED&atom=%2Fcmajo%2F11%2F1%2FE180.atom) [Web of Science](http://www.cmajopen.ca/lookup/external-ref?access_num=000278276700009&link_type=ISI) 30. Schultz SE, Rothwell DM, Chen Z, et al. (2013) Identifying cases of congestive heart failure from administrative data: a validation study using primary care patient records. Chronic Dis Inj Can 33:160–6. [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.24095/hpcdp.33.3.06&link_type=DOI) [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=23735455&link_type=MED&atom=%2Fcmajo%2F11%2F1%2FE180.atom) 31. Gilbert T, Neuburger J, Kraindler J, et al. (2018) Development and validation of a Hospital Frailty Risk Score focusing on older people in acute care settings using electronic hospital records: an observational study. Lancet 391:1775–82. [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1016/S0140-6736(18)30668-8&link_type=DOI) [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fcmajo%2F11%2F1%2FE180.atom) 32. Meyer M, Schwarz T, Renkawitz T, et al. (2021) Hospital Frailty Risk Score predicts adverse events in revision total hip and knee arthroplasty. Int Orthop 45:2765–72. 33. Austin PC, Rothwell DM, Tu JV (2002) A comparison of statistical modeling strategies for analyzing length of stay after CABG surgery. Health Serv Outcomes Res Methodol 3:107–33. [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1023/A:1024260023851&link_type=DOI) 34. Almashrafi A, Elmontsri M, Aylin P (2016) Systematic review of factors influencing length of stay in ICU after adult cardiac surgery. BMC Health Serv Res 16:318. 35. Diab MS, Bilkhu R, Soppa G, et al. (2018) The influence of prolonged intensive care stay on quality of life, recovery, and clinical outcomes following cardiac surgery: a prospective cohort study. J Thorac Cardiovasc Surg 156:1906–15e3. [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fcmajo%2F11%2F1%2FE180.atom) 36. Tran DTT, Tu JV, Dupuis J-Y, et al. (2018) Association of frailty and long-term survival in patients undergoing coronary artery bypass grafting. J Am Heart Assoc 7:e009882. 37. Sun LY, Eddeen AB, Mesana TG (2021) Disability-free survival after major cardiac surgery: a population-based retrospective cohort study. CMAJ Open 9:E384–93. [Abstract/FREE Full Text](http://www.cmajopen.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NToiY21ham8iO3M6NToicmVzaWQiO3M6ODoiOS8yL0UzODQiO3M6NDoiYXRvbSI7czoyMToiL2NtYWpvLzExLzEvRTE4MC5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 38. Sun LY, Rodger J, Duffett L, et al. (2021) Derivation of patient-defined adverse cardiovascular and noncardiovascular events through a modified Delphi process. JAMA Netw Open 4:e2032095. 39. Austin PC (2009) Using the standardized difference to compare the prevalence of a binary variable between two groups in observational research. Commun Stat Simul Comput 38:1228–34. [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1080/03610910902859574&link_type=DOI) 40. Rubin DB, Schenker N (1991) Multiple imputation in health-care databases: an overview and some applications. Stat Med 10:585–98. [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1002/sim.4780100410&link_type=DOI) [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=2057657&link_type=MED&atom=%2Fcmajo%2F11%2F1%2FE180.atom) [Web of Science](http://www.cmajopen.ca/lookup/external-ref?access_num=A1991FG07000008&link_type=ISI) 41. Kuk D, Varadhan R (2013) Model selection in competing risks regression. Stat Med 32:3077–88. [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1002/sim.5762&link_type=DOI) [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=23436643&link_type=MED&atom=%2Fcmajo%2F11%2F1%2FE180.atom) 42. Harrell FE Jr., Lee KL, Mark DB (1996) Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med 15:361–87. [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1002/(SICI)1097-0258(19960229)15:4<361::AID-SIM168>3.0.CO;2-4&link_type=DOI) [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=8668867&link_type=MED&atom=%2Fcmajo%2F11%2F1%2FE180.atom) [Web of Science](http://www.cmajopen.ca/lookup/external-ref?access_num=A1996TY77400003&link_type=ISI) 43. Shahian DM, O’Brien SM, Filardo G, et al., Society of Thoracic Surgeons Quality Measurement Task Force (2009) The Society of Thoracic Surgeons 2008 cardiac surgery risk models: part 1 — coronary artery bypass grafting surgery. Ann Thorac Surg 88(Suppl):S2–22. [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1016/j.athoracsur.2009.05.053&link_type=DOI) [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=19559822&link_type=MED&atom=%2Fcmajo%2F11%2F1%2FE180.atom) [Web of Science](http://www.cmajopen.ca/lookup/external-ref?access_num=000267469100002&link_type=ISI) 44. Ivanov J, Borger MA, Rao V, et al. (2006) The Toronto Risk Score for adverse events following cardiac surgery. Can J Cardiol 22:221–7. [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1016/S0828-282X(06)70900-X&link_type=DOI) [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=16520853&link_type=MED&atom=%2Fcmajo%2F11%2F1%2FE180.atom) [Web of Science](http://www.cmajopen.ca/lookup/external-ref?access_num=000235944200007&link_type=ISI) 45. Hannan EL, Farrell LS, Wechsler A, et al. (2013) The New York risk score for in-hospital and 30-day mortality for coronary artery bypass graft surgery. Ann Thorac Surg 95:46–52. [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1016/j.athoracsur.2012.08.047&link_type=DOI) [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=23200237&link_type=MED&atom=%2Fcmajo%2F11%2F1%2FE180.atom) 46. Tu JV, Jaglal SB, Naylor CD (1995) Multicenter validation of a risk index for mortality, intensive care unit stay, and overall hospital length of stay after cardiac surgery. Steering Committee of the Provincial Adult Cardiac Care Network of Ontario. Circulation 91:677–84. [Abstract/FREE Full Text](http://www.cmajopen.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MTQ6ImNpcmN1bGF0aW9uYWhhIjtzOjU6InJlc2lkIjtzOjg6IjkxLzMvNjc3IjtzOjQ6ImF0b20iO3M6MjE6Ii9jbWFqby8xMS8xL0UxODAuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 47. Triana AJ, Vyas R, Shah AS, et al. (2021) Predicting length of stay of coronary artery bypass grafting patients using machine learning. J Surg Res 264:68–75. 48. Alshakhs F, Alharthi H, Aslam N, et al. (2020) Predicting postoperative length of stay for isolated coronary artery bypass graft patients using machine learning. Int J Gen Med 13:751–62. 49. Oakes G (2018) Report on adult cardiac surgery: isolated coronary artery bypass graft (CABG) surgery isolated aortic valve replacement (AVR) surgery and combined CABG and AVR surgery: October 2011– March 2016 (CorHealth Ontario, North York). 50. Panah M, Andres LA, Strope SA, et al. (1998) Morbidity associated with prolonged hospital length of stay following cardiac surgery. Anesth Analg 86:50SCA. 51. Nakano M, Nomura Y, Suffredini G, et al. (2020) Functional outcomes of frail patients after cardiac surgery: an observational study. Anesth Analg 130:1534–44. [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fcmajo%2F11%2F1%2FE180.atom) 52. Sun LY, Tu JV, Lee DS, et al. (2018) Disability-free survival after coronary artery bypass grafting in women and men with heart failure. Open Heart 5:e000911. [Abstract/FREE Full Text](http://www.cmajopen.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6Nzoib3BlbmhydCI7czo1OiJyZXNpZCI7czoxMToiNS8yL2UwMDA5MTEiO3M6NDoiYXRvbSI7czoyMToiL2NtYWpvLzExLzEvRTE4MC5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 53. O’Brien SM, Feng L, He X, et al. (2018) The Society of Thoracic Surgeons 2018 adult cardiac surgery risk models: Part 2 — statistical methods and results. Ann Thorac Surg 105:1419–28. [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1016/j.athoracsur.2018.03.003&link_type=DOI) [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fcmajo%2F11%2F1%2FE180.atom) 54. Lazar HL, Fitzgerald C, Gross S, et al. (1995) Determinants of length of stay after coronary artery bypass graft surgery. Circulation 92(Suppl):II20–4. [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1161/01.CIR.92.9.20&link_type=DOI) [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=7586408&link_type=MED&atom=%2Fcmajo%2F11%2F1%2FE180.atom) 55. Sepehri A, Beggs T, Hassan A, et al. (2014) The impact of frailty on outcomes after cardiac surgery: a systematic review. J Thorac Cardiovasc Surg 148:3110–7. [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1016/j.jtcvs.2014.07.087&link_type=DOI) [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=25199821&link_type=MED&atom=%2Fcmajo%2F11%2F1%2FE180.atom) 56. Lal S, Gray A, Kim E, et al. (2020) Frailty in elderly patients undergoing cardiac surgery increases hospital stay and 12-month readmission rate. Heart Lung Circ 29:1187–94. 57. Prina AM, Cosco TD, Dening T, et al. (2015) The association between depressive symptoms in the community, non-psychiatric hospital admission and hospital outcomes: a systematic review. J Psychosom Res 78:25–33. [CrossRef](http://www.cmajopen.ca/lookup/external-ref?access_num=10.1016/j.jpsychores.2014.11.002&link_type=DOI) [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=25466985&link_type=MED&atom=%2Fcmajo%2F11%2F1%2FE180.atom) 58. Fisher K, Griffith LE, Gruneir A, et al. (2021) Functional limitations in people with multimorbidity and the association with mental health conditions: baseline data from the Canadian Longitudinal Study on Aging (CLSA). PLoS One 16:e0255907. 59. Sun LY, Tu JV, Sherrard H, et al. (2018) Sex-specific trends in incidence and mortality for urban and rural ambulatory patients with heart failure in eastern Ontario from 1994 to 2013. J Card Fail 24:568–74. [PubMed](http://www.cmajopen.ca/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fcmajo%2F11%2F1%2FE180.atom) 60. Sun LY, Bader Eddeen A, Wijeysundera HC, et al. (2021) Derivation and validation of a clinical model to predict death or cardiac hospitalizations while on the cardiac surgery waitlist. CMAJ 193:E1333–40. [Abstract/FREE Full Text](http://www.cmajopen.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiY21haiI7czo1OiJyZXNpZCI7czoxMjoiMTkzLzM0L0UxMzMzIjtzOjQ6ImF0b20iO3M6MjE6Ii9jbWFqby8xMS8xL0UxODAuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) * © 2023 CMA Impact Inc. or its licensors