European Heart Journal Advance Access originally published online on February 16, 2006
European Heart Journal 2006 27(9):1124; doi:10.1093/eurheartj/ehi799
An unusual meta-regression approach with a high potential of misleading conclusions
Institute for Clinical Epidemiology
Hebelstrasse 10
University Hospital Basel
4031 Basel
Switzerland
E-mail address: kollerm{at}uhbs.ch
Center for Medical Decision Making
Department of Public Health
Erasmus MC-University Medical Center Rotterdam
PO Box 1738
3000 DR Rotterdam
The Netherlands
Yu et al.1 systematically reviewed the literature of heart failure disease management programmes (DMPs) with the aim to identify crucial programme characteristics in reducing clinical outcomes for elderly patients with heart failure. The authors dichotomized individual trials as ‘effective’ or ‘ineffective’ based on whether a trial reported a statistically significant result or not. The authors determined the programme components (e.g. exercise) to be crucial if these were more prevalent in effective trials than in ineffective ones, although a clear-cut definition of ‘crucial’ is not given in the paper. We dispute the validity of the conclusions because of the naïve methodological approach.
Yu et al.1 implicitly performed an unusual type of meta-regression analysis. The correct way would have been to regress DMP characteristics against the effect size, e.g. a log relative risk using random-effects meta-regression.2,3 The dichotomization of trial results based on statistical significance poses several problems. First, the sample size determines the likelihood that a trial is labelled effective. Trials with ineffective DMPs may simply have accrued too low numbers of patients.4,5 Secondly, information on between-trial and within-trial variation needed for heterogeneity assessment and weighting of trials is discarded. Because of dichotomization, the authors used implicitly a naïve fixed-effect meta-regression that is not appropriate for complex DMPs with potential residual heterogeneity.3
The authors classified 12 trials as effective and nine as ineffective. Then, they assessed 47 characteristics (Table 5) for its relation to programme success. Although the authors refrain from performing 47 formal statistical tests against 12 successes and nine failures, they declare crucial characteristics as those which are more prevalent in effective trials, disregarding that at least some associations could be due to chance alone. Had the authors endeavoured to carry out 47 statistical tests, the issue of multiple testing and data dredging would have been more obvious. The problem of this approach can be illustrated with the component ‘education’. Education was equally often used in ineffective programmes as in effective programmes. Consequently, this component seems not to be associated with success. Because nobody puts this key component into question, this example shows the inappropriateness of the analysis. Even worse, the in-hospital phase seems negatively associated with effective trials (Table 5), although claimed in the discussion as an integral part of the DMP. Thus, components labelled crucial might not be based on the results and are likely to be identified by chance.2 As a result, Yu et al. suggest complex interventions, including the need to involve cardiologists directly, educate and integrate cardiac nurses, and implement exercise training and psychosocial counselling. We doubt that many of these crucial components would stand a randomized head-to-head comparison.
In conclusion, data dredging and inappropriate methodology are major pitfalls in this analysis. We would have liked to see an explicitly formulated research question with pre-specified covariates.2 Instead, Yu et al.1 aimed to identify crucial DMP characteristics from a large amount of possible factors. The results should be treated with great caution and might at its best be seen as exploratory. The findings should neither lead to the implementation of unwarranted complex and expensive programmes6 nor prevent the future research on important and well-designed head-to-head comparisons.
References
- Yu DS, Thompson DR, Lee DT. Disease management programmes for older people with heart failure: crucial characteristics which improve post-discharge outcomes. Eur Heart J doi:10.1093/eurheartj/ehi656.
- Thompson SG, Higgins JP. How should meta-regression analyses be undertaken and interpreted? Stat Med 2002; 21: 1559–1573.[CrossRef][Web of Science][Medline]
- Higgins JP, Thompson SG. Controlling the risk of spurious findings from meta-regression. Stat Med 2004; 23: 1663–1682.[CrossRef][Web of Science][Medline]
- Cline CM, Israelsson BY, Willenheimer RB, Broms K, Erhardt LR. Cost effective management programme for heart failure reduces hospitalisation. Heart 1998; 80: 442–446.
[Abstract/Free Full Text] - Jaarsma T, Halfens R, Huijer Abu-Saad H, Dracup K, Gorgels T, van Ree J, Stappers J. Effects of education and support on self-care and resource utilization in patients with heart failure. Eur Heart J 1999; 20: 673–682.
[Abstract/Free Full Text] - Boyd CM, Darer J, Boult C, Fried LP, Boult L, Wu AW. Clinical practice guidelines and quality of care for older patients with multiple comorbid diseases: implications for pay for performance. JAMA 2005; 294: 716–724.
[Abstract/Free Full Text]
CiteULike 
Connotea 
Del.icio.us What's this?
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||