|
Can Observational Studies Be Used for Supporting New Treatments?
(Thursday, June 15, 2023) Observational studies such as collection and evaluation of Real World Data (RWD) collected from medical records and other uncontrolled sources could play a crucial role in advancing medical knowledge by shedding light on disease incidence, prevalence, natural history, prognosis, and aiding in the development of clinical risk scores. However, when it comes to evaluating the effectiveness of interventions like surgeries, drugs, medical devices, or radiotherapy, observational studies have limitations in establishing cause-and-effect relationships. A recent perspective in Journal of American Medical Association describes ways where observational studies can be designed to limit bias. There are two kinds of observational studies, retrospective review, or prospectively planned review of medical records. In either design, one way is to enhance the reliability of causal inferences in observational research is to require investigators to establish clear enrollment criteria and rules and predefine endpoints and null hypothesis akin to a randomized clinical trial. Another method is early separation of Kaplan-Meier curves. The Kaplan-Meier survival curve represents the time-to-event endpoints and maximizes the use of each participant's time-related data. In this analysis, participants contribute to the survival estimate until the event of interest occurs or until they are censored. If Kaplan-Meier curves show significant separation at the beginning of the follow-up period for interventions that typically require a longer time to show effects, it suggests confounding factors that bias the conclusions. Certain interventions may not immediately demonstrate their efficacy. For instance, vaccines may take time to elicit an immune response, and surgical interventions aimed at reducing long-term complications may not yield observable differences for weeks. However, in some cases, observational studies exhibit a clear separation of Kaplan-Meier curves within the initial day of follow-up, which is biologically implausible. This phenomenon indicates the presence of residual confounding rather than a true effect. Examples include the association of bisphosphonates with mortality and the impact of geriatrician involvement following serious traumatic injury. The authors show examples where bias in observation is obvious. In an observational study examining the efficacy of COVID-19 booster vaccinations, the Kaplan-Meier curves started to separate on day 8 after vaccination, which seems implausibly early given the typical immune response timeframe. Similarly, retrospective comparisons between chimeric antigen receptor T-cell (CAR-T) therapy and standard salvage chemotherapy showed survival differences from day 0, contradicting the delivery method, action, and associated risks of CAR-T therapy. Another study comparing left atrial appendage occlusion devices with direct oral anticoagulation revealed Kaplan-Meier curves separating from day 0, despite the expected gradual reduction in stroke risk over time. These early separations likely indicate confounding factors that bias the results of the observational studies. One limitation of using Kaplan-Meier curves to detect selection bias in observational studies is that it requires knowledge of the disease's pathophysiology, natural history, and expected intervention efficacy. Additionally, published observational studies often lack Kaplan-Meier curves, relying instead on tables displaying relative risks or odds ratios between patient groups, which fail to capture changes in these measures over time. To identify biases and imbalances, studies investigating time-to-event endpoints should incorporate graphical representations of the data. When interventions appear to yield results too quickly, it is essential to approach the study findings with caution. The observed outcomes may be too good to be true, necessitating a critical evaluation of potential confounding factors and biases. By employing rigorous methodologies and addressing limitations, researchers can improve the validity and reliability of observational studies. 
AUTHOR
Dr. Mukesh Kumar Founder & CEO, FDAMap Email: mkumar@fdamap.com Linkedin: Mukesh Kumar, PhD, RAC Instagram: mukeshkumarrac Twitter: @FDA_MAP Youtube: MukeshKumarFDAMap |
|
|
|
