ABSTRACT
Background: Risk stratification models are used to assess the risk of death in surgery. Objective: To conduct a critical analysis of the EuroSCORE logistic model (ES) application in 2,692 patients undergoing Coronary Artery Bypass Grafting (CABG) in four public hospitals in the Rio de Janeiro Municipality, from 1999 through to December 2003. Methods: Random samples of 150 medical records for surviving and deceased patients were selected at four public hospitals in the City of Rio de Janeiro. The ES was applied, using the logistical model. The observed lethality rate and that forecast by the model were compared. The measurement of the discriminatory power was estimated by the area under the ROC curve. Results: 546 of the 600 selected medical records were located. A significant difference was noted between the prevalence rates for the risk factors in the Brazilian and European populations. The forecast lethality rate was 3.62% (CI-95%: 3.47-3.78) while the estimated observed rate was 12.22% (CI-95%- 10.99-13.46). In all risk ranges, the predicted lethality rate is under-estimated, with notable differences between the predicted and observed rates. The area under the ROC curve was estimated at 0.62. Conclusion: The differences in the prevalence rates for the risk factors constituting the ES, associated with its low power of discrimination, hamper any recommendation of the use of this model in Brazil, without the necessary adjustments.
RESUMO
FUNDAMENTOS: Modelos de estratificação de risco são utilizados em cirurgia para avaliar risco de morte. OBJETIVO: Fazer análise crítica da aplicação do EuroSCORE (ES) em amostras de prontuários de 2692 pacientes submetidos à cirurgia de revascularização miocárdica em 4 hospitais públicos do município do Rio de Janeiro, no período de 1999 a 2003. MÉTODOS: Foram selecionadas, em quatro hospitais públicos da cidade do Rio de Janeiro, amostras aleatórias de 150 prontuários de pacientes por hospital, sobreviventes e óbitos. Aplicou-se o ES utilizando-se o modelo logístico. A letalidade observada e prevista pelo modelo foi comparada. A aferição do poder discriminante foi estimada pela área sob a curva ROC. RESULTADOS: Localizados 546 dos 600 prontuários selecionados. Observou-se significativa diferença entre prevalências dos fatores de risco entre nossa população e européia. Letalidade prevista foi 3,62% (IC-95%: 3,47-3,78) e observada estimada foi 12,22% (IC-95%- 10,99-13,46). Em todas as faixas de risco, há subestimação da letalidade prevista, com diferenças notáveis entre prevista e observada.Área sob a curva ROC foi estimada em 0,62. CONCLUSÃO: Diferenças das prevalências dos fatores de risco que compõem o ES associado ao baixo poder discriminatório desaconselham a utilização do modelo em nosso meio sem devidos ajustes.
INTRODUCTION
Risk stratification models are used to assess the risk of death in heart surgery[1]. The intention is to support the assessment of the balance of risks and benefits of surgical intervention, and also the adequacy of resource allocation[2]. Carvalho et al.[3] evaluated 23 preoperative factors in patients undergoing coronary artery bypass graft (CABG) in public hospitals in the city of Rio de Janeiro. Age over 70 years, current smoking, hypertension, dyslipidemia, stroke, injury of isolated left main coronary artery greater than 50% and association between injury of left main coronary artery with coronary lesions in any other coronary artery have a statistical association with death as the outcome[3].
Hospital mortality is a health system quality indicator. It should be analyzed along with the study of influence of other variables on surgical outcomes[4]. Multivariate risk assessment models can be useful in comparing the performance of institutions or even surgical teams, although they may be inadequate to predict individual risk[1,3].
Sets of risk scores for death and occurrence of complications were prepared in patients undergoing CABG. RioEscore[5], NECDSG[6] and EuroSCORE[7] are some examples. EuroSCORE was created using a European database of 19,030 patients undergoing surgery at 128 institutions in eight different countries from September to November 1995. Preoperative variables totaling 97 were collected, and associations with the outcome in-hospital deaths were sought, including those occurring in patients who remained hospitalized for more than 30 days after intervention. After bivariate analysis and logistic regression, 17 factors were selected for the final logistic model with the aim of assessing the risk of death[7].
EuroSCORE was used in several population groups in Europe[8], Japan[9] and USA[10]. Li et al.[11] demonstrated that the EuroSCORE applied to a series of 9248 patients that underwent CABG procedures, 87.8% of them overestimated mortality. Yap et al.[12] considered the model unsuitable to predict the risk of death for CABG when applied to 5592 patients at six Australian institutions from July 2001 to July 2005. Gogbashian et al.[13] published a systematic review of six studies in which the EuroSCORE was prospectively applied. They concluded there are suggestive evidences that the performance of the additive EuroSCORE overestimates mortality in patients with low EuroSCORE (
<6) and underestimated mortality in those with higher EuroSCORE (>13).
Due to these variables results in applying the EuroSCORE in different populations, trying to apply the same in Brazil would be considerable. The objective is to perform a critical analysis of the implementation of the EuroSCORE in samples of 2692 patients undergoing CABG in four public hospitals in Rio de Janeiro, from January 1999 to December 2003.
METHODS
This work is part of the "Lethality in highly complex procedures in ischemic heart disease in the state of Rio de Janeiro", sponsored by the Science Resource Foundation of Rio de Janeiro (FAPERJ), the human resources of the Federal University of Rio de Janeiro (UFRJ) and Oswaldo Cruz Foundation (Fiocruz).
The CABGs were identified from the Hospital Admission Orders database with the corresponding codes of the Brazilian Unified Health System (SUS) in Rio de Janeiro, from January 1999 to December 2003, excluding those situations where there was also valve intervention. The final selection only included the latest CABG in each individual and four hospitals where 96% of these surgeries were performed in the city. We identified 2692 patients operated in four hospitals, including two University Hospitals and two Specialized Hospitals in cardiology , each pair consisting of a State and Federal Hospital , designated A,B,C and D.
A) Sample selection
The ideal analysis plan to compare survivors and death victims would be one in which the samples of these two groups were equivalent. To this end, equivalent samples of both groups should have been selected in each hospital after CABG in order to obtain 150 individuals per hospital. As the death toll was less than 75 individuals in at least three hospitals (A,C and D), all the dead were selected according to the information in the Hospital Admission Orders. Sufficient amount of survivors was also selected through simple random sampling in these hospitals in order to complete the quota of 150 individuals per hospital. In hospital B, two equivalent samples were randomly selected: 75 survivors and 75 dead. The random sample selections of patients were performed by using the routine sample Stata statistical software[14]. Table 1 lists the quantities of selected patients and operated per hospitals, and to estimate the mortality rate the individuals data were weighted in inverse probability of selection in the sample.
Data collection was retrospectively made in 2006 through medical charts stored in the hospitals, collected by trained cardiologists in standardized forms, which included socio-demographic data, hospital admissions, risk factors, comorbidities, complementary exams, prescriptions, surgery conditions, angioplasty procedures, prior CABG, postoperative complications and hospital evolution. The collection form recorded 499 variables.
The data collected were transferred to electronic forms on the program Epidata version 3.1[15]. The definitions of variables and their classification criteria have been described in instruction sheet for consultation by researchers in order to standardize the filling. The project was submitted to the Ethics Committee in Research of Clementino Fraga Filho Hospital, which belongs to the Federal University of Rio de Janeiro, and approved (Protocol 102/05).
b) Calculation of the presumed lethality with EuroSCORE
The presumed mortality was calculated by assuming the EuroSCORE logistic model: where β
0 is the constant and β
i is the regression coefficients of variablesχi. The coefficients β
i for each variable and its formula application were described by Nashef et al.[16].
Death was considered at any time as occurred during the hospital stay related to the CABG, or within 30 days from surgery to those patients who were discharged. Data were obtained by consulting the Death Records. This consultation was possible due to the study by Godoy et al.[17], who used the method of probabilistic linkage between the Hospital Admission Orders databases and death certificates, which were part of the same global project that belongs to this study.
Chronic obstructive pulmonary disease (COPD), extracardiac arteriopathy, neurological dysfunction, critical preoperative state and unstable angina are factors requiring adjustments in the definitions to make them similar to the original definitions of the EuroSCORE study[7] (Table 1). Other variables, such as other surgeries besides CABG, thoracic aorta surgery and surgery for infective endocarditis were not applied because they were grounds for exclusion of patients in our study. The medical charts did not record the condition of emergency surgery, however, this variable could not have been measured. In our study, the risk factors were considered present when noted in medical charts. The absence was demonstrated by its negative record or omission of information. Chart 1 lists some factors evaluated in the EuroSCORE[7], but discarded in the final model.
Chart 1 - Definitions of the factors evaluation in the EuroSCORE and Rio de Janeiro Studies.
The prevalence and risk factors of the population in our study were estimated in order to compare them with those observed in the EuroSCORE study[7]. The t-student test was applied to assess the significance of the difference between the mean ages. The chi-square distribution was used to indicate the differences among the prevalence of the other variables listed in Table 1. Differences have been considered significant when the P value was less than 0.05 (5%).
The presumed lethalities were calculated according to the logistic model in three ranges of additive EuroSCORE, and their respective confidence intervals of 95%. The performance of the EuroSCORE was assessed by comparing the observed mortality with the assumptions of the logistic model. The area under the ROC (receiver operating characteristic) was estimated to measure the discriminating power of this model[18,19].
RESULTS
It was spotted 546 of the 600 medical charts selected for the research. The prevalence of risk factors of the EuroSCORE study population and ours are listed in Table 2. All differences among prevalences were significant. The estimated age distribution of our sample shows a higher prevalence of young patients under 60 years old, and less of elderly patients aged 70 years or more. The prevalence of diabetes and arterial hypertension were approximately twice as high in Rio de Janeiro as in the population of the EuroSCORE.
Mean age and standard deviation in years were 62.5 ± 10.7 and 60.8 ± 9.6 (P < 0.0005) for the populations of the EuroSCORE and Rio de Janeiro, respectively.
Table 3 shows the distributions of presumed lethality, according to the logistic model for risk ranges of additive EuroSCORE and the estimated observed mortality in patients from Rio de Janeiro. The presumed mortality was assumed 3.62% (95% CI 3.47 to 3.78) and the observed lethality estimation was 12.22% (95% CI: 10.99 to 13.46) for all patients in Rio de Janeiro. In all risk ranges there is an underestimation of presumed mortality when compared with the observed ones. The degree of underestimation is inversely correlated with the degree of risk range.
Table 4 shows the distribution of the presumed lethality by the logistic EuroSCORE and the observed mortality estimated in hospital with patients from Rio de Janeiro. The underestimations of the presumed compared to observed lethality were remarkable, mainly in hospital B. In the case of this hospital, the divergence between the assumed and observed mortality results in reversing the relative position of that hospital mortality relating to others (Table 4).
The confrontation between the assumed and observed lethality by the EuroSCORE resulted in an area under the ROC curve of 0.60 with a confidence interval of 95% between 0.56 and 0.65, indicating a low discriminatory power of the patient population operated Rio de Janeiro (Figure 1).
Fig. 1 - Area under the ROC curve of 0.60 with a confidence interval of 95% between 0.56 and 0.65, indicating a low discriminatory power of the model in the patient population operated in Rio de Janeiro
DISCUSSION
The introduction of models to predict risk of death allows cardiologists and cardiac surgeons to speculate on the complications probability and death of candidate patients for CABG[20]. In 2004, the ACC/AHA Guideline Update for Coronary Artery Bypass Graft Surgery[21] recommended the use of risk models for patients and doctors to assess the risk and benefit of the indication for CABG. This recommendation is based on the validation of different scoring systems, especially the EuroSCORE, a system considered by Gogbashian et al.[13] as the model to predict better risk established and further validated in a systematic review conducted in which were included series of patients from Japan, Belgium, France, Turkey and United Kingdom. On the other hand, studies with patients in Australia[12] and China[11] reported flaws in the model when applied to these populations, suggesting caution in its use and recommended the creation of new models adapted to the specific realities of each country.
Moraes et al.[22] retrospectively evaluated the applicability of the EuroSCORE in 759 patients undergoing CABG at the Heart Institute of Pernambuco, during 2003 and 2004 in Brazil. The authors estimated the accuracy of the model at 69.9%, considering the model reasonable when applied to that population. Unlike our study, it was reported the whole approaches to identify the 17 models factors, not having observed a statistically significant difference between the assumed and observed mortality, however, adjustments were made to explain this similarity.
EuroSCORE was developed from data collected from 19,030 patients who underwent surgery in 128 health systems in eight European countries. Sixty-eight preoperative risk factors and 29 operative were analyzed to assess its association with death[7]. The statistical model used was the multiple logistic regression. The variables of bivariate association with death resulted in P-value less than 20%, which were selected for the model, and those in which the P-value was above 0.05 were eliminated. The discriminatory power of the model for those who survived or evolved to death was assessed by the area under the ROC curve of 0.79, a satisfactory level. Logistic model was used because it showed the highest reliability in predicting the death when prospectively applied in 24 026 patients operated between January 1999 and April 2004[23].
Differences were statistically significant and relevant between the prevalence of 15 common risk factors in the sets used in the original study and in our study (Table 2). Our patients were younger on average. The positive occurrence of five variables was not observed in any patient, two of them having been ground for exclusion. It is necessary to emphasize that in our study the information loss[3] in variables such as COPD, extracardiac arteriopathy and critical preoperative state were relevant, over 10%. By admitting that, the uninformed variables would be considered absent as their presence may have been underestimated. Considering that this is a retrospective study partly available to measure some factors in the medical charts, both the scores in our study and the presumed lethality may have been underestimated by applying the model.
According to our study and the population of the EuroSCORE[7], the differences among the prevalence of factors related to the stage of ischemic disease were relevant, such as acute myocardial infarction (AMI): 15.2% and 9.7%, and unstable angina, 39.1% and 8.0% respectively. These findings may connote greater severity in patients from our series, and the prevalence of hypertension and diabetes in Rio de Janeiro were considerably higher than the population of the EuroSCORE[7], which may explain the higher incidence of neurological dysfunction in our population (Table 2). Despite the apparent severity in the patients, the occurrences were lower in situations that demanded urgent attention as reflected by the critical pre-operatory condition. This observation may represent significant differences among health systems. Similar differences were observed by other authors when applied the EuroSCORE in 5592 patients undergoing CABG in Australia[12].
The absence of emergency registration to perform the surgery was unexpected. The difference between the presumed lethality, 3.62% (95% CI 3.47 to 3.78) and the observed lethality, 12.22% (95% CI: 10.99 to 13.46), may be due to the lack of information on this variable (Table 3). The prevalence of emergency surgery factor in the population of the EuroSCORE was 4.9%. In order to overcome the influence of this variable in our population, two hypothetical prevalence for this condition, 25 and 100% were simulated. In the first case, the presumed lethality would be 4.57% (95% CI: 4.36 to 4.7%), nearly three times lower than the observed lethality. In the second, with a prevalence of 100% of emergency surgeries, the lethality rate would be 6.84% (95% CI 6.58 to 7.10%), about half of the observed mortality. Therefore, it is highly unlikely that all patients had been operated in emergency, yet the EuroSCORE logistic model would have underestimated the mortality by 44%.
The differences between the presumed and observed lethalities in the three stages of risk recommend caution in the use of the EuroSCORE in our midst. The hierarchy between the presumed lethality is the same for the observed one, according to the risk ranges. However, there are notable discrepancies between the presumed and observed lethality in each risk category, according to the additive EuroSCORE (Table 3).
Our series of patients is limited to those treated in public hospitals in the state capital of Rio de Janeiro. The information scarcity to the performance of the private health system in Brazil, does not allow us to discuss the performance of the model in surgical patients in private hospitals. Gomes et al.[5] studied a cohort of 1458 patients operated in a private hospital identified as C, from June 2000 to March 2003. However, it was used its own model, called RioScore, with different variables from those used in the EuroESCORE.
The low predictive power of the EuroSCORE in our population when compared with other series[8-12] may derive from its inadequate implementation. In our study, EuroSCORE was retrospectively applied with adaptations to six factors and instances of positive factors that generate meaningful scores were not found when calculating the probability of death, such as emergency surgery, pulmonary arterial hypertension and ruptured interventricular septum after infarction[16]. The institutional factor may be a possible influence on the model performance. In a previous publication[3], this factor was associated with mortality. The difference between the health services in Rio de Janeiro in relation to the European's may have influenced the poor performance of the model.
The clinical data used was investigated during consultation on medical charts. Despite the information losses already mentioned, the data reliability was assumed to be higher than the administrative one[24] used by other authors in applying the EuroSCORE in non-European populations[10].
Another source of differences between lethality estimation is the variability in defining the period of occurrence of death. This definition sometimes includes the observation of patients within 30 days after hospital discharge, or during hospital stay related to the surgery at any time, as it is suggested in our study. However, in other studies[2,8,11,22,25,26] are considered deaths only those occurred during hospitalization. EuroSCORE was applied on several occasions in which the reports are unclear about the definition of the period used in the analysis on the occurrence of death[2,8,11,22,25,26].
The comparison between the presumed and observed lethality per hospital shows large differences (Table 4). Hospital B, with the presumption of the lowest mortality, had the highest lethality rate. Hospitals C and D, specialized in cardiology, are a subset with similar presumed and observed lethality. Hospital A has the highest presumed lethality, showing a higher prevalence of factors which is score for EuroSCORE. Its observed lethality is similar to hospital B, both being university general hospitals. These observations make us understand that the institution has an influence on the model performance.
The area under the ROC curve summarizes the model performance by displaying the ability to discriminate between survivors and death victims, representing the probability that an individual is adequately stratified by the model according to the outcome[18]. The ROC curves are useful to compare two or more models, or applying a model to different sets. Despite the reference to a good performance of EuroSCORE in the series previously cited, such as European populations[8], the United States[10], Asia[9.11] and Australia[12], in which the areas under the ROC curve are above 0.70, our results indicated a low discriminatory power, with an area under the curve of 0.61 (95% CI 0.56 to 0.65)(Figure 1).
When applying the EuroSCORE in our sample significant differences were found between the prevalence of 17 risk factors and two factors not included in the mortality estimation. If they are coupled with the low discriminatory power of the model in our set, its use as an operative risk assessment tool is unadvisable in patients who may be undergoing CABG in public hospitals in Rio de Janeiro. The weight adjustment factor could probably correct its performance and make it an applicable tool to our system of care. However, it is also likely that these weights are being modified over time and are still dependent on environmental conditions, where the surgeries are performed in the institutions.
REFERENCES
1. Asimakopoulos G, Al-Ruzzeh S, Ambler G, Omar RZ, Punjabi P, Amrani M, et al. An evaluation of existing risk stratification models as a tool for comparison of surgical performances for coronary artery bypass grafting between institutions. Eur J Cardiovasc Surg. 2003;23(6):935-41.
2. Pitkänen O, Niskanen M, Rehnberg S, Hippeläinen M, Hynynen M. Intra-institutional prediction of outcome after cardiac surgery: comparison between a locally derived model and the EuroSCORE. Eur J Cardiothoracic Surg. 2000;18(6):703-10.
3. Carvalho MRM, Souza e Silva NA, Klein CE. Associação de fatores pré-operatórios e óbitos na cirurgia de revascularização miocárdica em hospitais públicos do Rio de Janeiro:1999-2003. Rev SOCERJ. 2008;21(5):311-9.
4. Pons JM, Espinas JA, Borras JM, Moreno V, Martin I, Granados A. Cardiac surgical mortality: comparison among different additive risk-scoring models in a multicenter sample. Arch Surg. 1998;133(10):1053-7. [MedLine]
5. Gomes RV, Tura B, Mendonça Filho HTF, Campos LAA, Rouge A, Nogueira PMM, et al. RioEscore: escore preditivo de mortalidade para pacientes submetidos à cirurgia cardíaca baseado em variáveis de pré, per e primeiro dia de pós-operatório. Rev SOCERJ. 2005;18(6):516-26.
6. O'Connor GT, Plume SK, Olmstead EM, Coffin LH, Morton JR, Maloney CT, et al. A regional prospective study of inhospital mortality associated with coronary artery bypass grafting. The Northern New England Cardiovascular Disease Study Group. JAMA. 1991;266(6):803-9. [MedLine]
7. Roques F, Nashef SA, Michel P, Gauducheau E, de Vincentiis C, Baudet E, et al. Risk factors and outcome in European cardiac surgery: analysis of the EuroSCORE multinational database of 19030 patients. Eur J Cardiothorac Surg. 1999;15(6):816-22.
8. Roques F, Nashef SA, Michel P, Pinna Pintor P, David M, Baudet E, et al. Does EuroSCORE work in individual European countries? Eur J Cardiothorac Surg. 2002;18(1):27-30.
9. Kawachi Y, Nakashima A, Toshima Y, Arinaga K, Kawano H. Evaluation of the quality of cardiovascular surgery care using risk stratification analysis according to the EuroSCORE additive model. Circ J. 2002;66(2):145-8. [MedLine]
10. Nashef SA, Roques F, Hammill BG, Peterson ED, Michel P, Grover FL, et al. Validation of European System for Cardiac Operative Risk Evaluation (EuroSCORE) in North American cardiac surgery. Eur J Cardiothorac Surg. 2002;22(1):101-5. [MedLine]
11. Zheng Z, Li Y, Zhang S, Hu S; Chinese CABG Registry Study. The Chinese coronary artery bypass grafting registry study: how well does the EuroSCORE predict operative risk for Chinese population? Eur J Cardiothorac Surg. 2009;35(1):54-8. [MedLine]
12. Yap CH, Reid C, Yii M, Rowland MA, Mohajeri M, Skillington PD, et al. Validation of the EuroSCORE model in Australia. Eur J Cardiothorac Surg. 2006;29(4):441-6.
13. Gogbashian A, Sedrakyan A, Treasure T. EuroSCORE: a systematic review of international performance. Eur J Cardiothorac Surg. 2004;25(5):695-700. [MedLine]
14. Stata Corporation. Stata Version 7. College Station: Stata Corporation;2005.
15. Lauritsen JM. EpiData data entry, data management and basic statistical analysis system. Odense:EpiData Association;2000-2008. Disponível em: http://www.epidata.dk.
16. Nashef SAM. Letter to the editor. Eur J Cardiothorac Surg. 2003; 24:1-2.
17. Godoy PH, Oliveira GMM, Souza e Silva NA, Klein CH. Diferença nas taxas de letalidade e nas principais causas de óbito, entre homens e mulheres, por revascularização miocárdica cirúrgica. Rev SOCERJ. 2008;21(3):311-9.
18. Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology. 1982;143(1):29-36. [MedLine]
19. Vanagas G. Receiver operating characteristic curves and comparison of cardiac surgery risk stratification systems. Int Cardiovasc Thorac Surg. 2004;3(2):319-22.
20. Cortina Romero JM. Criteria for using risk models in cardiac surgery. Rev Esp Cardiol. 2008;61(6):567-71. [MedLine]
21. Eagle KA, Guyton RA, Davidoff R, Edwards FH, Ewy GA, Gardner TJ; American College of Cardiology; American Heart Association Task Force on Practice Guidelines; American Society for Thoracic Surgery and the Society of Thoracic Surgeons, et al. ACC/AHA 2004 guideline update for coronary artery bypass graft surgery: summary article: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines (Committee to Update the 1999 Guidelines for Coronary Artery Bypass Graft Surgery). Circulation. 2004;110(9):1168-76. [MedLine]
22. Moraes F, Duarte C, Cardoso E, Tenório E, Pereira V, Lampreia D, et al. Avaliação do EuroSCORE como preditor de mortalidade em cirurgia de revascularização miocárdica no Instituto do Coração de Pernambuco. Rev Bras Cir Cardiovasc. 2006;21(1):29-34. View article
23. Zingone B, Pappalardo A, Dreas L. Logistic versus additive EuroSCORE. A comparative assessment of the two models in an independent population sample. Eur J Cardiothoracic Surg. 2004;26(6):1134-40.
24. Herbert MA, Prince SL, Williams JL, Magee MJ, Mack MJ. Are unaudited records from an outcomes registry database accurate? Ann Thorac Surg. 2004;77(6):1960-4.
25. Lafuente S, Trilla A, Bruni L, González R, Bertrán MJ, Pomar JL, et al. Validation of the EuroSCORE probabilistic model in patients undergoing coronary bypass grafting. Rev Esp Cardiol. 2008;61(6):589-94. [MedLine]
26. Karabulut H, Toraman F, Alhan C, Camur G, Evrenkaya S, Dagdelen S, et al. EuroSCORE overestimates the cardiac operative risk. Cardiovasc Surg. 2003;11(4):295-8. [MedLine]
Science Resource Foundation of Rio de Janeiro (FAPERJ)
Article receive on Friday, December 18, 2009