Reporting in surgical research

Summary
Transparent and standardized reporting in clinical research is a prerequisite for optimal healthcare decision-making. This applies to any clinical discipline, but may be specifically challenging, when complex surgical interventions are involved. There, the impact of surgeons’ experience on the size of treatment effects, standardization of interventions and of outcome assessment need to be specifically addressed.
In a previous investigation on selective outcome reporting in surgical trials (Rosenthal and Dwan 2013), we found a remarkably high percentage of unexplained discrepancies between registry entries and final reports of randomized controlled trials (RCTs) that were published in high-impact surgical journals. In order to address problems of reporting in surgical research more in depth, I decided to dedicate my PhD to this topic. Thus, the overall aim of my PhD research is to investigate and promote transparent and standardized reporting in surgical research. As follows, I outline the different aspects I specifically addressed.
Project 1: How to write a surgical clinical research protocol: literature review and practical guide
Any clinical research starts with asking a research question. The question shall be embedded in the existing body of evidence. If the study question and hypothesis is not precise and the study is not designed in a sound manner, this will impact on reporting and overall study conclusions. The core document of the design phase is the study protocol. Our first project was thus to develop a straightforward 10-step practice guideline on how to develop a surgical clinical trial protocol with a focus on methodological aspects. Our project and initiative was highly welcomed by the journal reviewer of the American Journal of Surgery, where the study was published (Rosenthal et al. 2014) and is reflected in his reviewer comment “This is a valuable report that describes the steps necessary to optimize the design of clinical research studies. This information is important and the article should be required reading for all surgeons who are embarking on a research career.”
Project 2: The use of systematic reviews when designing and reporting surgical trials
A systematic review (SR) on the existing literature should be part of any clinical research project to justify the planned research from a scientific, ethical and economic point of view as well as to inform the trial design and, finally to sum up the results within the existing body of evidence. In a meta-epidemiological research project we investigated to what extend investigators systematically searched the literature in the context of their project and whether this information was provided in the final publication (Rosenthal et al. 2015a). Of 596 studies, 51 RCTs published in 3 high-impact general surgical journals were identified. SRs were referenced in 65%, either to summarize evidence concerning a related topic (43%) or the study topic (excluding first-in-area RCTs: introduction 24%, discussion 26%). No SRs were used to inform trial design and no trial updated a SR for the integration of the new results. In conclusion, in the surgical literature SR are rarely used to inform trial design, justify the research and synthesize knowledge for informed decision making.
Project 3: Completion and publication rates of surgical randomized controlled trials – an empirical study
In a next step, we addressed important issues when conducting and reporting results of clinical trials in surgery. We investigated to what extent surgical trials were discontinued early and/or not published. For this purpose, all RCT protocols approved from 2000 to 2003 by six ethics committees in Canada, Germany and Switzerland were screened. We explored risk factors for early trial discontinuation due to slow recruitment and compared surgical to medical trials. Early trial discontinuation due to slow recruitment has an impact on reporting: first, overall trial conclusions due to limited power may be biased if the targeted sample size has not been reached and second early discontinuation may increase the risk of publication bias if results typically derived from underpowered studies are not published.
In total, 863 RCT protocols involving adult patients were identified; 127 in surgery (15%) and 736 in medicine (85%). Surgical trials were discontinued for any reason more often than medical trials (43% versus 27%, risk difference 16% (95% confidence interval [CI] 5%, 26%); p=0.001) and were also more often discontinued for slow recruitment (18% versus 11%, risk difference 8% (95% CI 0.1%, 16%); p=0.020). The percentage of trials not published as full journal article was similar in surgical and medical trials (44% versus 40%, risk difference 4% (95% CI -5%, 14%); p=0.373). Discontinuation of surgical trials was a strong risk factor for non-publication (odds ratio 4.18, 95% CI 1.45, 12.06; p=0.008) (Rosenthal et al. 2015b).
Project 4: Reporting of adverse events in surgical trials: critical appraisal of current practice
Reporting of surgical outcomes should include detailed reporting of harm. The information needs to be generated according to uniform and reproducible standards that allow for the comparison of results, techniques, centres or surgeons. For postoperative complications, several classifications have been proposed and validated; however, there exists no recognised classification system for intraoperative complications. Therefore, we aimed at assessing the current practice of reporting intra- and postoperative complications in surgical trials that were published in 2010 in three major surgical journals (Rosenthal et al. 2015c). We identified 46 trials that reported intra- and postoperative complications. These complications were reported separately in 42% and pooled in 15%. In 37% intraoperative, in 2% postoperative, and in 4% both intra- and postoperative complications were not reported at all. Exact definitions were provided in 13% for intraoperative and in 50% for postoperative complications. A classification was used in 9% for intra- and in 54% for postoperative complications, most frequently according to severity. As further research, we thus plan to develop and validate a classification of intraoperative complications in order to facilitate the evaluation of safety and the continuous quality control of surgical interventions with the ultimate goal to contribute to patient safety.
Project 5: Definition and Classification of Intraoperative Complications (CLASSIC): Delphi Study and pilot evaluation
Our abovementioned investigation (Project 4) of reporting of harm clearly showed the urgent need for a classification of intraoperative complications. We therefore conducted a two-stage Delphi study among experts in surgical practice and trial methodology to develop a definition and classification of intraoperative complications (Rosenthal et al. 2015d).
In the Delphi study, a total of 40 out of 52 experts (77% return rate) from 14 countries took part in both rounds. It resulted in a comprehensive definition of intraoperative complications and a straightforward classification. We then conducted a pilot study reviewing 60 records involving surgical interventions of variable complexity by two independent reviewers. It showed good practicability (6 on a 7-point scale) and an 87% agreement with a weighted kappa of 0.83 (95% CI 0.73, 0.94) and an intraclass correlation coefficient of 0.83 (95% CI 0.73, 0.90).
Project 6: How to report multiple outcome metrics in virtual reality simulation
Virtual reality (VR) simulation is increasingly being used for assessment and training purposes in various surgical disciplines. However, there is no consensus on how to report outcomes derived from simulators. VR simulators objectively measure multiple outcomes, which is frequently handled by selective reporting or multiple testing. This compromises comparison between different VR studies and introduces a risk of bias. Thus, suitable methods are needed to either address multiplicity issues or combine evidence from multiple possibly related outcomes into a lower dimensional outcome. We developed an algorithm for summarizing multiple VR outcome metrics into a total score and illustrate the approach with two real data examples (Rosenthal et al. 2015e). Given the increasing number of articles in the field, a standardized and transparent approach for analyzing and reporting VR outcome data is of utmost importance to enhance the validity of VR derived surgical reports.
Project 7 (ongoing): Disregarding paired data in surgical research – evaluation of current practice and estimation of implications on study results
In surgical practice and research, patients may be affected by a pathology and treated on both sides of their body, e.g. in extremity surgery. As a consequence, paired data are generated. In the context of our consultancy service for surgeons we were faced with studies that did not account for the paired design in the statistical analysis. We therefore decided to carry out a systematic literature review based on the example of inguinal hernia repair, one of the most frequently conducted surgical interventions (ongoing). The aim of this project is to first evaluate to what extent paired data are accounted for in the statistical analysis and second to estimate the impact on overall study conclusions if this specific design issue is not correctly considered in the statistical analysis.
Conclusion
In conclusion, with the present work the reporting of surgical trials is addressed from two points of view: i) evaluation of current practice and ii) proposal for procedures to address identified challenges in reporting. Since any study report depends on the study conception and conduct, these two phases are included in the evaluation.