Marcos Gradim TiveronI; Helton Augusto BomfimII; Maycon Soto SimplícioI; Marcos Henriques BergonsoII; Milena Paiva Brasil de MatosI; Sergio Marques FerreiraI; Eraldo Antônio PellosoI; Rubens Tofano de BarrosIII
CCS: Canadian Cardiovascular Society
Eurosc1: The European system for cardiac operative risk evaluation
Eurosc2: The European system for cardiac operative risk evaluation II
NYHA: New York Heart Association
ROC: Receiver Operating Characteristic
SPSS: Statistical Package for the Social Sciences
SP-SCORE: São Paulo System for Cardiac Operative Risk Evaluation
STS: Society of Thoracic Surgeons score
The search for quality excellence in the delivery of health services requires, in addition to technical and structural capacity, a broad knowledge of patients regarding the severity and complexity of their disease in order to identify the factors responsible for the results and long-term prognosis. Among medical specialties, cardiovascular surgery is one of the most complex. Its results are dependent of pre-, intra- and post-operative variables and are assessed by risk scores already developed that estimate the morbidity and mortality of surgical procedure.
The use of performance indicators has emerged as an objective way to measure the quality of health services. Risk stratification serves to inform patients and healthcare professionals about the potential risk of complications or death for the group of individuals with similar risk profile who had undergone the procedure proposed. The creation and validation of local models become increasingly necessary for demographic, socio-economic and cultural differences found in our environment and the need for external validation of existing prognostic models is of paramount importance for their applicability in clinical practice situations. For this, every proposed method should be calibrated and discriminated, or that is, presenting good accuracy and distinguish patients at low and high risk, respectively[4,5]. The importance of external validation of the risk model is not only limited in clinical application. It also serves to recognize the limitations and structural deficiencies of different medical institutions and strategies aiming to qualify these medical services.
Among the most commonly used risk scores are the Society of Thoracic Surgeons score (STS score), The European system for cardiac operative risk evaluation (EuroSCORE) and the EuroSCORE II. Recently, Mejía et al., created and validated internally at the Heart Institute of the Clinics Hospital, Faculty of Medicine, University of São Paulo (Incor/HC-USP) a risk model, called InsCor, having meant to be a tool of easy implementation and good accuracy for prognostic analysis of patients undergoing valve replacement with or without CABG in our country. However this model has not been evaluated in any institution outside the place of origin.
The aim of the study was to apply the InsCor in patients undergoing CABG and/or valve surgery at Santa Casa de Marília, a reference center for public patients in the state of São Paulo and compare its performance with the STS, EuroSCORE (Eurosc1) and EuroSCORE II (Eurosc2).
The present study is a cohort. It is a prospective, observational, analytical, single-center study and performed at the Center for Cardiac Surgery in Marília. Between April 2011 and June 2013, patients older than 18 years undergoing coronary artery bypass grafting, valve surgery, and the association of the two procedures, including reoperations were included consecutively, totaling 562 patients. The exclusion criterion was patients under 18 years of age and surgical indication for any other reason. Of the 562 patients, 26 patients (4.63%) died. 368 (65.5%) CABG, 160 (28.5%) valve surgery and 34 (6%) associated coronary and valve were performed.
Data were collected from an institutional database stored in an Excel software spreadsheet that already includes all variables of STS score, logistic EuroSCORE, EuroSCORE II and InsCor respecting their definitions (Chart 1), which allowed the calculation of the EuroSCORE II and InsCor after their publications. The calculations of the STS score, Eurosc1 and Eurosc2 were performed by own application from a smartphone and in the case of InsCor from a graph with its own scoring system. The follow-up was limited to hospital phase, with the primary outcome of hospital mortality that included the period between surgery and discharge. The estimated mortality was calculated from the scores STS score, Eurosc1 and Eurosc2 and IS.
The calibration was calculated using the Hosmer-Lemeshow test and the P value>0.05 indicates that the model fits the data and predicts mortality appropriately. Discrimination distinguishes patients at low and high risk, and is measured by the area under the ROC curve (Receiver Operating Characteristics). Statistical analysis was performed using SPSS software version 16.0 for Windows (IBM Corporation Armonk, New York). The performance of models was measured by comparing observed and expected mortality in risk groups established by models. The Fisher exact test was used for contingency tables. The P value <0.05 was considered significant. This study was approved by the Research Ethics Committee of the Faculty of Medicine of Marília under number 767 329 on August 27, 2014.
Performance results of the STS score, EuroSCORE, EuroSCORE II and InsCor
In the analysis of the whole group, association was observed between the STS score and death (P<0.001). The Hosmer-Lemeshow test showed a goodness-of-fit statistic=8.9696 with 8 degrees of freedom and P=0.3449. For coronary surgeries, we observe that the STS score was not associated with death (P=0.182) but showed good calibration (P=0.210). For valve surgery, we found that it is associated with death (P=0.009) with good calibration (P=0.460). Associated surgeries (coronary and valve), was not associated with death (P=0.4078), but showed good calibration (P=0.2648).
In the analysis of the whole group, association was observed between Eurosc1 and death (P<0.001). The Hosmer-Lemeshow test showed a goodness-of-fit statistic = 4.9246 with 8 degrees of freedom and P=0.7656. For coronary procedures, we found that the EuroSCORE is associated with death (P=0.021) with good calibration (P=0.529). For valve surgery, we found that the model is associated with death (P=0.004) in addition to good calibration (P=0.893). For associated surgeries (coronary and valve), although the Eurosc1 was not associated with death (P=0.1033), it showed good calibration (P=0.2911).
In the analysis of the whole group, association was observed between Eurosc2 and death (P<0.001). The Hosmer-Lemeshow test showed a goodness-of-fit statistic = 9.8963 with 8 degrees of freedom and P=0.2724. For coronary procedures, we observed that Eurosc2 is associated with death (P=0.040) and demonstrates good calibration (P=0.250). For valve surgery, we found that the model is associated with death (P<0.001) and demonstrates good calibration (P=0.423). For associated surgeries (coronary and valve), although the Eurosc2 was not associated with death (P=0.5159), it showed good calibration (P=0.5659).
In the analysis of the whole group, association was observed between InsCor and death (P<0.001). The Hosmer-Lemeshow test showed a goodness-of-fit statistic = 8.9533 with 4 degrees of freedom and P=0.0623. For coronary procedures, we observe that although the InsCor was not associated with death (P=0.059), it showed good calibration (P=0.110). For valve surgery, we observed that InsCor is associated with death (P=0.002) and good calibration (P=0.743). For associated surgeries (coronary and valve), the InsCor was associated with death (P=0.022) but the calibration was not adequate (P<0.001).
Discrimination results (Figure 1)
According to the discriminative power of STSscore, we observe that the area under the ROC curve was 0.649 (95% CI 0.529-0.770, P=0.012) (Table 1).
According to the discriminative power of Eurosc1, we observe that the area under the ROC curve was 0.706 (95% CI 0.589-0.823, P<0.001) (Table 1).
According to the discriminative power of Eurosc2, we observe that the area under the ROC curve was 0.704 (95% CI 0.590-0.818, P=0.001) (Table 1).
According to the discriminative power of InsCor, we observe that the area under the ROC curve was 0.739 (95% CI 0.638 to 0.839, P<0.001) (Table 1).
The present study is the first assessing the performance of InsCor outside the institution that developed it. We apply and compare the most current and popular scores worldwide and found little difference when comparing the EuroSCORE with EuroSCORE II, a result similar to other international studies.
The good current medical practice requires doctors and hospitals proposing to perform cardiovascular surgery to assess their patients adequately and in a responsible manner. In this preoperative evaluation, calculation and estimation of complications or death are required and able to propose strategies in the search for better results.
The prognostic models for risk analysis are inefficient if they serve only to the population where they were developed, thus they must have good performance in other populations, or that is, they must be validated externally.
The EuroSCORE system that for nearly 15 years provides an estimate of postoperative mortality in cardiac surgery is widely used to have good accuracy and adequate discriminating power in different countries. In our midst, Andrade et al. analyzed 804 patients undergoing valve surgery at the Heart Institute of Pernambuco and obtained a good fit with an area under the ROC curve of 0.731 (95% CI 0.660 to 0.793 P<0.001). This same effectiveness has been demonstrated by other authors who studied patients undergoing CABG and found good accuracy, estimated at 69.9% and 89.2%[8,9]. On the other hand, Carvalho et al. analyzed 546 patients undergoing CABG and observed underestimation of lethality expected, with notable differences between the predicted and observed with an area under the ROC curve estimated at 0.62 advising against the use of the model in our midst. This fact is explained by differences in prevalence among risk factors of the model and the differences found in the population studied. An adjustment to the weight of the factors that composes such model could correct its performance and make it an applicable tool. In our study, we found that the EuroSCORE was associated with death, with good calibration for CABG and heart valve surgeries and proper ROC curve of 0.706 (95% CI 0.589 to 0.823, P<0.001). According to the analysis of discrimination by ROC curve we have a poor discrimination with ROC<0.70, acceptable with ROC<0.75, good with ROC<0.80 and very good with ROC>0.80.
On October 2011, Nashef et al. showed the remodeled EuroSCORE that became known as the EuroSCORE II, only logistical and started using new variables as: calculation of creatinine clearance, insulin dependent diabetes, NYHA class and CCS class 4, in addition to reclassification of the following variables: ejection fraction, pulmonary hypertension, urgency of the procedure and type of procedure performed. In calibration, the observed mortality was 3.9% and the expected mortality by EuroSCORE II of 3.77%, compared to 4.6% of the original EuroSCORE. Barili et al. validated the EuroSCORE II from the retrospective analysis of the results of a database of three institutions containing 12,325 consecutive patients and reaching a hospital mortality rate of 2.2% and a high discriminative power with an area under the ROC of 0.82 (95% CI: 0.80-0.85) and good calibration curve to a mortality of 30% predicted. However, it did not appear to significantly improve the performance of older versions in the highest tertiles of risk. Paparella et al., studied data on 6191 patients and found hospital mortality rate of 4.85% and EuroSCORE II of 4.40±7.04%. The area under the ROC curve of 0.83 showed good discriminative ability. In the analysis of calibration, there was an underestimation of the expected mortality in high-risk patients. In South America, Borracci et al., performed the validation of the EuroSCORE II in 503 patients undergoing cardiac surgery and obtained a hospital mortality rate of 4.17% with predicted mortality of 3.18% (P=0.402). The area under the ROC curve was 0.85 (P=0.0001) and the model showed good calibration in predicting in-hospital mortality (Hosmer-Lemeshow, P=0.082). The EuroSCORE II showed a good overall discriminative ability and calibration in this population, however, the performed model underestimates in-hospital mortality of patients with lower risk. In Brazil, the first validation of the EuroSCORE II was performed by Lisboa et al., from the analysis of 1000 consecutive patients undergoing coronary and/or valve surgery.
In calibration, the Hosmer-Lemeshow test was inadequate (P=0.0003). However, in the discrimination, the area under the ROC curve was 0.81 [95% CI (0.76-0.85), P<0.001], concluding that the EuroSCORE II became more complex and similar to the international literature with respect to poor calibration to predict mortality in patients undergoing coronary and/or valve in our midst, reinforcing the importance of a local model. In the present study, we found that the EuroSCORE II was associated with death, showing good calibration in CABG and valve surgery performed with proper ROC curve area of 0.704 (95% CI, 0.590 to 0.818, P=0.001).
The model of surgical risk created by the American Society of Thoracic Surgeons (Society of Thoracic Surgeons-STS) allows the calculation of mortality and morbidity, including rates of complications such as the risk of prolonged hospitalization, stroke, prolonged intubation, renal failure, deep wound infection and reoperation in patients undergoing CABG, valve (replacement or repair) and the association between the two types of surgery. In our environment, Ikeoka et al. evaluated the STS score and achieved good calibration and discrimination for mortality and morbidity with an area under the ROC curve of 0.76 and 0.75 (P<0.001) respectively in a group of patients undergoing CABG. In study similar to ours, Wang et al., Australia, analyzed four models of risk (EuroSCORE, EuroSCORE II, STS score, and a local score called AusScore) for patients undergoing CABG. The estimated 30-day mortality by EuroSCORE was 2.8% (1.6 to 5.2%), EuroSCORE II 1.6% (from 1.0 to 2.8%), STS Score of 2.3% (1.3 - 4.5%) and the AusScore 0.5% (0.2 to 1.1%). Regarding the EuroSCORE, the EuroSCORE II, the STS score and the AusScore showed a slight improvement in calibration but similar for 30-day mortality discrimination. In our study, although the STS score was not associated with death, it showed good calibration in division by groups and the ROC curve of 0.649 was poor (95% CI 0.529 to 0.770, P=0.012).
In our study, we found overall hospital mortality of 4.6%. Regarding mortality and expected by the scores observed in the three groups the analysis shows an underestimation of mortality estimated by three international scores, especially Eurosc1 and Eurosc2. When comparing the study by Wang et al., we observed a wide variation in the risk of patients seen in our service, which could be explained by delayed access to health care and/or late surgical indication of patients to the surgical procedure, a system of reflexes of poor health. In patients undergoing CABG, mortality was 3.2%, ranging from 1.0% to 4% (0.8%-72%). In valve surgery, mortality was 6.2%, ranging from 1.8% to 9.6% (0.8% -59%). In the combined surgery group, mortality was 11.7%, ranging from 2.8% to 12.5% (from 0.6 to 26%) (Table 2).
In 2013, a model of national risk developed from the analysis of 3000 patients undergoing coronary artery bypass and/or valve at the Heart Institute of the Clinics Hospital, Faculty of Medicine, University of São Paulo, called of InsCor was proposed by Mejia et al., a remodeling product of two models (2000 Bernstein-Parsonnet and EuroSCORE) validated and faced with our reality and has become an important tool for patients treated at InCor- HCFMUSP. 10 variables were selected: age >70 years; female; surgical coronary revascularization + valve; myocardial infarction < 90 days; reoperation; surgical treatment of aortic valve; surgical treatment of tricuspid valve; creatinine > 2 mg/dL; ejection fraction < 30%; and events (use of preoperative inotropic support, cardiogenic shock, cardiac resuscitation, use of intra-aortic balloon, acute renal failure, cardiac massage, tracheal intubation, tachycardia or ventricular fibrillation). The Hosmer-Lemeshow test was 0.184, indicating excellent calibration and the area under the ROC curve was 0.79 (95% from 0.74 to 0.83, P<0.001 IC). In our study, the InsCor was not associated with mortality for CABG, presented good calibration for CABG and heart valve surgeries and had adequate ROC of 0.739 (95% CI, 0.638 - 0.839, P<0,001).
This study assessed and compared three scores of international risk, being an American and two European and one developed in our country. The Brazilian experience in creating risk scores nationwide is completing nearly a decade[20,21], but without external validation. The search for simple models containing few variables are of wider applicability, since they contain the significant risk factors and have an appropriate degree of prediction because they are easier to be incorporated in patient care[22-24]
We can consider limitations of the study the fact that it is a single center and with a small sample. It is an ongoing multicenter study in referral hospitals in the state of São Paulo, aiming at reshaping the InsCor for creation of the SCORE-SP (São Paulo System for Cardiac Operative Risk Evaluation). This project will initiate the São Paulo State Registry of Cardiovascular Surgery, in order to improve the quality of results and the safety of patients undergoing cardiac surgery in the State of São Paulo.
In conclusion, the InsCor was the best model to predict the final outcome of survival or mortality, followed by EuroSCORE. The EuroSCORE was the best model in adjusted mortality for CABG and InsCor, along with the EuroSCORE II for valve surgery. Therefore, the local model InsCor showed good accuracy, in addition to being effective and easy to apply, mainly by using a smaller number of variables compared to the other models, variables that represent relevant risk factors in our population.
1. Mejía OAV. Predição de mortalidade em cirurgia de coronária e/ou valva no InCor: Validação de dois modelos externos e comparação com o modelo desenvolvido localmente (InsCor). [Tese de Doutorado]. São Paulo: Universidade de São Paulo, 2012. 118p.
2. Nashef SA, Roques F, Michel P, Gauducheau E, Lemeshow S, Salamon R. European system for cardiac operative risk evaluation (EuroSCORE). Eur J Cardiothorac Surg. 1999;16(1):9-13. [MedLine]
3. Bleeker SE, Moll HA, Steyerberg EW, Donders AR, Derksen-Lubsen G, Grobbee DE, et al. External validation is necessary in prediction research: a clinical example. J Clin Epidemiol. 2003;56(9):826-32. [MedLine]
4. Altman DG, Vergouwe Y, Royston P, Moons KG. Prognosis and prognostic research: validating a prognostic model. Research Methods & Reporting. BMJ. 2009;338:b605. [MedLine]
5. Mejía OAV, Lisboa LAF, Dallan LAO, Pomerantzeff PMA, Moreira LFP, Jatene FB, et al. Validação do 2000 Bernstein-Parsonnet e EuroSCORE no Instituto do Coração - USP. Rev Bras Cir Cardiovasc. 2012;27(2):187-94. [MedLine] View article
6. Mejía OAV, Lisboa LAF, Puig LB, Moreira LFP, Dallan LAO, Pomerantzeff PMA, et al. InsCor: um método simples e acurado para avaliação do risco em cirurgia cardíaca. Arq Bras Cardiol. 2013;100(3):246-54. [MedLine]
7. Andrade ING, Moraes Neto FR, Oliveira JPSP, Silva ITC, Andrade TG, Moraes CRR. Avaliação do EuroSCORE como preditor de mortalidade em cirurgia cardíaca valvar no Instituto do Coração de Pernambuco. Rev Bras Cir Cardiovasc. 2010;25(1):11-8. [MedLine] View article
8. Sá MPBO, Soares EF, Santos CA, Figueiredo OJ, Lima ROA, Escobar RR, et al. EuroSCORE e mortalidade em cirurgia de revascularização miocárdica no Pronto Socorro Cardiológico de Pernambuco. Rev Bras Cir Cardiovasc. 2010;25(4):474-82. [MedLine] View article
9. Moraes Neto F, Duarte C, Cardoso E, Tenório E, Pereira V, Lampreia D, et al. Avaliação do EuroSCORE como preditor de mortalidade em cirurgia de revascularização miocárdica no Instituto do Coração de Pernambuco. Rev Bras Cir Cardiovasc. 2006;21(1):29-34. View article
10. Carvalho MRM, Souza e Silva NA, Klein CH, Oliveira GMM. Aplicação do EuroSCORE na cirurgia de revascularização miocárdica em hospitais públicos do Rio de Janeiro. Rev Bras Cir Cardiovasc. 2010;25(2):209-17. [MedLine] View article
11. Nashef SA, Roques F, Sharples LD, Nilsson J, Smith C, Goldstone AR, et al. EuroSCORE II. Eur J Cardiothorac Surg. 2012;41(4):734-44.
12. Barili F, Pacini D, Capo A, Rasovic O, Grossi C, Alamanni F, et al. Does EuroSCORE II perform better than its original versions? A multicentre validation study. Eur Heart J. 2013;34(1):22-9. [MedLine]
13. Paparella D, Guida P, Di Eusanio G, Caparrotti S, Gregorini R, Cassese M, et al. Risk stratification for in-hospital mortality after cardiac surgery: external validation of EuroSCORE II in a prospective regional registry. Eur J Cardiothorac Surg. 2014;46(5):840-8. [MedLine]
14. Borracci RA, Rubio M, Celano L, Ingino CA, Allende NG, Ahuad Guerrero RA. Prospective validation of EuroSCORE II in patients undergoing cardiac surgery in Argentinean centres. Interacte Cardiovasc Thorac Surg. 2014;18(5):539-43.
15. Lisboa LAF, Mejia OAV, Moreira LFP, Dallan LAO, Pomerantzeff PMA, Dallan LRP, et al. EuroSCORE II e a importância de um modelo local, InsCor e o futuro SP-SCORE. Rev Bras Cir Cardiovasc 2014;29(1):1-8. [MedLine]
16. Shahian DM, O'Brien SM, Filardo G, Ferraris VA, Haan CK, Rich JB, et al; Society of Thoracic Surgeons Quality Measurement Task Force. The Society of Thoracic Surgeons 2008 cardiac surgery risk models: part 1--coronary artery bypass grafting surgery. Ann Thorac Surg. 2009;88(1 Suppl):S2-22. [MedLine]
17. Ikeoka DT, Fernandes VA, Gebara O, Garcia JCT, Barros e Silva PGM, Rodrigues MJ, et al. Avaliação dos escores da Society of Thoracic Surgeons para cirurgia de revascularização miocárdica isolada em uma população brasileira. Rev Bras Cir Cardiovasc. 2014;29(1):51-8. [MedLine]
18. Wang TK, Li AY, Ramanathan T, Stewart RA, Gamble G, White HD. Comparison of four risk scores for contemporary isolated coronary artery bypass grafting. Heart Lung Circ. 2014;23(5):469-74. [MedLine]
19. Bernstein AD, Parsonnet V. Bedside estimation of risk as an aid for decision-making in cardiac surgery. Ann Thorac Surg. 2000;69(3):823-8. [MedLine]
20. Gomes RV, Tura B, Mendonça Filho HTF, Campos LAA, Rouge A, Nogueira PMM, et al. RIOEscore: escore preditivo de mortalidade para pacientes submetidos à cirurgia cardíaca baseado em variáveis de pré, per e primeiro dia de pós-operatório. Rev SOCERJ. 2005;18(6):516-26.
21. Cadore MP, Guaragna JCVC, Anacker JFA, Albuquerque LC, Bodanese LC, Piccoli JCE, et al. Proposição de um escore de risco cirúrgico em pacientes submetidos à cirurgia de revascularização miocárdica. Rev Bras Cir Cardiovasc 2010;25(4):447-56. [MedLine] View article
22. Jones RH, Hannan EL, Hammermeister KE, DeLong ER, O'Connor GT, Luepker RV, et al. Identification of preoperative variables needed for risk adjustment of short-term mortality after coronary artery bypass graft surgery. The Working Group Panel on the Cooperative CABG Database Project. J Am Coll Cardiol. 1996;28(6):1478-87. [MedLine]
23. Omar RZ, Ambler G, Royston P, Eliahoo J, Taylor KM. Cardiac surgery risk modeling for mortality: a review of current practice and suggestions for improvement. Ann Thorac Surg. 2004;77(6):2232-7. [MedLine]
24. Ranucci M, Castelvecchio S, Menicanti L, Frigiola A, Pelissero G. Accuracy, calibration and clinical performance of the EuroSCORE: can we reduce the number of variables? Eur J Cardiothorac Surg. 2010;37(3):724-2.
25. Mejía OAV, Lisboa LAF, Dallan LAO, Pomerantzeff PMA, Trindade EM, Jatene FB, et al. Heart surgery programs innovation using surgical risk stratification at the São Paulo State Public Healthcare System: SP-SCORE-SUS study. Rev Bras Cir Cardiovasc. 2013;28(2):263-9. [MedLine] View article
No financial support.
Authors' roles & responsibilities
MGT: Analysis and/or interpretation of data, final approval of the manuscript conception and study design, conduct of operations, and/or experiments, writing of the manuscript or revising it critically for its content
HAB: Performing the procedures and/or experiments
MSS: Performing the procedures and/or experiments
MHB: Performing the procedures and/or experiments
MPBM: Performing the procedures and/or experiments
SMF: Performing the procedures and/or experiments
EAP: Final approval of manuscript
RTB: Final approval of the manuscript, performing the operations and/or experiments, writing of the manuscript or review of its content
Article receive on Tuesday, May 27, 2014