COVID-19 Policy Impact Evaluation: A guide to common design issues.
American journal of epidemiology
Policy responses to COVID-19, particularly those related to non-pharmaceutical interventions, are unprecedented in scale and scope. However, policy impact evaluations require a complex combination of circumstance, study design, data, statistics, and analysis. Beyond the issues that are faced for any policy, evaluation of COVID-19 policies is complicated by additional challenges related to infectious disease dynamics and a multiplicity of interventions. The methods needed for policy-level impact evaluation are not often used or taught in epidemiology, and differ in important ways that may not be obvious. Methodological complications of policy evaluations can make it difficult for decision-makers and researchers to synthesize and evaluate strength of evidence in COVID-19 health policy papers. We (1) introduce the basic suite of policy impact evaluation designs for observational data, including cross-sectional analyses, pre/post, interrupted time-series, and difference-in-differences analysis, (2) demonstrate key ways in which the requirements and assumptions underlying these designs are often violated in the context of COVID-19, and (3) provide decision-makers and reviewers a conceptual and graphical guide to identifying these key violations. The overall goal of this paper is to help epidemiologists, policy-makers, journal editors, journalists, researchers, and other research consumers understand and weigh the strengths and limitations of evidence.
View details for DOI 10.1093/aje/kwab185
View details for PubMedID 34180960
Association of Convalescent Plasma Treatment With Clinical Outcomes in Patients With COVID-19: A Systematic Review and Meta-analysis.
Importance: Convalescent plasma is a proposed treatment for COVID-19.Objective: To assess clinical outcomes with convalescent plasma treatment vs placebo or standard of care in peer-reviewed and preprint publications or press releases of randomized clinical trials (RCTs).Data Sources: PubMed, the Cochrane COVID-19 trial registry, and the Living Overview of Evidence platform were searched until January 29, 2021.Study Selection: The RCTs selected compared any type of convalescent plasma vs placebo or standard of care for patients with confirmed or suspected COVID-19 in any treatment setting.Data Extraction and Synthesis: Two reviewers independently extracted data on relevant clinical outcomes, trial characteristics, and patient characteristics and used the Cochrane Risk of Bias Assessment Tool. The primary analysis included peer-reviewed publications of RCTs only, whereas the secondary analysis included all publicly available RCT data (peer-reviewed publications, preprints, and press releases). Inverse variance-weighted meta-analyses were conducted to summarize the treatment effects. The certainty of the evidence was assessed using the Grading of Recommendations Assessment, Development, and Evaluation.Main Outcomes and Measures: All-cause mortality, length of hospital stay, clinical improvement, clinical deterioration, mechanical ventilation use, and serious adverse events.Results: A total of 1060 patients from 4 peer-reviewed RCTs and 10?722 patients from 6 other publicly available RCTs were included. The summary risk ratio (RR) for all-cause mortality with convalescent plasma in the 4 peer-reviewed RCTs was 0.93 (95% CI, 0.63 to 1.38), the absolute risk difference was -1.21% (95% CI, -5.29% to 2.88%), and there was low certainty of the evidence due to imprecision. Across all 10 RCTs, the summary RR was 1.02 (95% CI, 0.92 to 1.12) and there was moderate certainty of the evidence due to inclusion of unpublished data. Among the peer-reviewed RCTs, the summary hazard ratio was 1.17 (95% CI, 0.07 to 20.34) for length of hospital stay, the summary RR was 0.76 (95% CI, 0.20 to 2.87) for mechanical ventilation use (the absolute risk difference for mechanical ventilation use was -2.56% [95% CI, -13.16% to 8.05%]), and there was low certainty of the evidence due to imprecision for both outcomes. Limited data on clinical improvement, clinical deterioration, and serious adverse events showed no significant differences.Conclusions and Relevance: Treatment with convalescent plasma compared with placebo or standard of care was not significantly associated with a decrease in all-cause mortality or with any benefit for other clinical outcomes. The certainty of the evidence was low to moderate for all-cause mortality and low for other outcomes.
View details for DOI 10.1001/jama.2021.2747
View details for PubMedID 33635310
Problems with Evidence Assessment in COVID-19 Health Policy Impact Evaluation (PEACHPIE): A systematic strength of methods review.
medRxiv : the preprint server for health sciences
The impact of policies on COVID-19 outcomes is one of the most important questions of our time. Unfortunately, there are substantial concerns about the strength and quality of the literature examining policy impacts. This study systematically assessed the currently published COVID-19 policy impact literature for a checklist of study design elements and methodological issues.We included studies that were primarily designed to estimate the quantitative impact of one or more implemented COVID-19 policies on direct SARS-CoV-2 and COVID-19 outcomes. After searching PubMed for peer-reviewed articles published on November 26 or earlier and screening, all studies were reviewed by three reviewers independently and in consensus. The review tool was based on review guidance for assessing COVID-19 health policy impact evaluation analyses, including first identifying the assumptions behind the methods used, followed by assessing graphical display of outcomes data, functional form for the outcomes, timing between policy and impact, concurrent changes to the outcomes, and an overall rating.After 102 articles were identified as potentially meeting inclusion criteria, we identified 36 published articles that evaluated the quantitative impact of COVID-19 policies on direct COVID-19 outcomes. The majority (n=23/36) of studies in our sample examined the impact of stay-at-home requirements. Nine studies were set aside due to inappropriate study design (n=8 pre/post; n=1 cross-section), and 27 articles were given a full consensus assessment. 20/27 met criteria for graphical display of data, 5/27 for functional form, 19/27 for timing between policy implementation and impact, and only 3/27 for concurrent changes to the outcomes. Only 1/27 studies passed all of the above checks, and 4/27 were rated as overall appropriate. Including the 9 studies set aside, we found that only four (or by a stricter standard, only one) of the 36 identified published and peer-reviewed health policy impact evaluation studies passed a set of key design checks for identifying the causal impact of policies on COVID-19 outcomes.The current literature directly evaluating the impact of COVID-19 policies largely fails to meet key design criteria for useful inference. This may be partially due to the circumstances for evaluation being particularly difficult, as well as a context with desire for rapid publication, the importance of the topic, and weak peer review processes. Importantly, weak evidence is non-informative and does not indicate how effective these policies were on COVID-19 outcomes.
View details for DOI 10.1101/2021.01.21.21250243
View details for PubMedID 33501457
View details for PubMedCentralID PMC7836129
Letter of concern regarding Reduction in COVID-19 infection using surgical facial masks outside the healthcare system.
Danish medical journal
2020; 67 (12)
View details for PubMedID 33269698
HIV self-testing among young women in rural South Africa: A randomized controlled trial comparing clinic-based HIV testing to the choice of either clinic testing or HIV self-testing with secondary distribution to peers and partners
2020; 21: 100327
HIV testing rates in many hyper-endemic areas are lower than needed to curtail the HIV epidemic. New HIV testing strategies are needed to overcome barriers to traditional clinic based testing; HIV self-testing is one modality that offers promise in reaching individuals who experience barriers to clinic-based testing.We conducted a randomized control trial among young women ages 18-26 living in rural Mpumalanga, South Africa where they were randomized in a 1:1 allocation to either the: (1) HIV Counseling and Testing (HCT) arm: an invitation to test at one of the 9 local government clinics where free HCT is provided and is standard of care (SOC), or (2) choice arm: choice of either a clinic-based HCT invitation or oral HIV Self-Testing (HIVST) kits. Depending on the arm, participants were also provided either: (1) 4 HCT invitations to provide to peers/partners for HIV testing at one of the 9 local clinics, or (2) 4 HIV self-test kits to provide to peers/partners (thus 5 total HIVST kits or HCT invitations). Young women were asked to return 3 months and 9 months after enrollment to assess testing uptake and invitation or kit distribution to peers and partners and experiences with testing. Peers and partners who were reported by index participants to have received kits/invitations during follow-up visits were also invited to attend a study visit to assess their testing experiences. The trial is registered at clinical trials.gov NCT03162965.287 young women were enrolled and randomized, with 146 randomized to the HCT arm and 141 to the choice (HCT or HIVST) arm. Of those randomized to the choice arm, over 95% (n=135) chose the HIV self-testing kit and only 6 individuals chose HCT. At the 3-month follow-up visit, 92% of index participants in the choice arm reported having tested for HIV compared to 43% of participants in the HCT arm, resulting in a significant risk difference of 49% (95% CI 40%, 58%). By 9 months, this difference decreased to a risk difference of 25% (95% CI 17%, 33%) between arms (96% in the choice arm and 72% in the HCT arm). Participants in the choice arm were also more likely to invite peers and partners to test compared to the HCT arm (94% vs. 76% or an average of 4.97 vs 2.79 tests). Few male partners were invited to test by index participants; however, index participants in the choice arm were more likely to have their male partners test than index participants in the HCT arm (RR 2.99, 95% CI 1.45, 6.16).When given a choice between clinic-based HIV testing and HIV oral self-testing, the overwhelming majority of young women chose HIVST. In addition, those offered a choice of HIV testing modality were much more likely to test, distribute test kits to peers and partners, and to have peers and partners who reported testing compared to the HCT arm. Self-testing offers an important opportunity to significantly increase testing rates among young women and their peers and partners compared to clinic-based HCT. Other strategies to reach men with testing are needed.US National Institutes of Health.
View details for DOI 10.1016/j.eclinm.2020.100327
View details for Web of Science ID 000646228100017
View details for PubMedID 32322811
View details for PubMedCentralID PMC7171186
HIV Care Continuum and Meeting 90-90-90 Targets: Cascade of Care Analyses of a U.S. Military Cohort.
INTRODUCTION: The new initiative by the Department of Health and Human Services (DHHS) aims to decrease new HIV infections in the U.S. by 75% within 5years and 90% within 10years. Our objective was to evaluate whether the U.S. military provides a good example of the benefits of such policies.MATERIALS AND METHODS: We conducted an analysis of a cohort of 1,405 active duty military personnel with HIV enrolled in the Natural History Study who were diagnosed between 2003 and 2015 at six U.S. military medical centers. The study was approved by institutional review boards at the Uniformed Services University of the Health Sciences and each of the sites. We evaluated the impact of Department of Defense (DoD) HIV care policies, including screening, linkage to care, treatment eligibility, and combined antiretroviral therapy (cART) initiation on achieving viral suppression (VS) within 3years of diagnosis. As a secondary outcome, we evaluated the DoD's achievement of UNAIDS 90-90-90 targets.RESULTS: Nearly all (99%) were linked to care within 60days. Among patients diagnosed in 2003-2009, 77.5% (95% confidence intervals (CI) 73.9-80.6%) became eligible for cART within 3years of diagnosis, 70.6% (95% CI 66.6-74.1%) overall initiated cART, and 64.2% (95% CI 60.1-68.0%) overall achieved VS. Among patients diagnosed in 2010-2015, 98.7% (95% CI 96.7-99.5%) became eligible for cART within 3years of diagnosis, 98.5% (95% CI 96.4-99.4%) overall initiated cART, and 89.8% (95% CI 86.0-92.5%) overall achieved VS.CONCLUSIONS: U.S. military HIV policies have been highly successful in achieving VS goals, exceeding the UNAIDS 90-90-90 targets. In spite of limitations, including generalizability, this example demonstrates the feasibility of the DHHS initiative to decrease new infections through testing, early treatment, and retention in care.
View details for DOI 10.1093/milmed/usaa021
View details for PubMedID 32207528
Limitations of the UNAIDS 90-90-90 metrics: a simulation-based comparison of cross-sectional and longitudinal metrics for the HIV care continuum.
AIDS (London, England)
OBJECTIVES: The UNAIDS 90-90-90 and other cross-sectional metrics can lead to potentially counterintuitive conclusions when used to evaluate health systems' performance. This study demonstrates how time and population dynamics impact UNAIDS 90-90-90 metrics in comparison with a longitudinal analogue.DESIGN: A simplified simulation representing a hypothetical population was used to estimate and compare inference from UNAIDS 90-90-90 metrics and a longitudinal metrics based on Kaplan-Meier-estimated 2-year probability of transition between stages.METHODS: We simulated a large cohort over 15 years. Everyone started out at risk for HIV, and then transitioned through the HIV care continuum based on fixed daily probabilities of acquiring HIV, learning status, entering care, initiating ART, and becoming virally suppressed, or dying. Within simulations we only varied the probability of ART initiation. We repeated the simulation with an increased probability of death.RESULTS: The cross-sectional probability of being on ART among persons who were diagnosed responded relatively slowly to changes in the rate of ART initiation. Increases in ART initiation rates caused apparent declines in the cross-sectional probability of being virally suppressed among persons who had initiated ART, despite no changes in the rate of viral suppression. In some cases, higher mortality resulted in the cross-sectional metrics implying improved healthcare system performance. The longitudinal continuum was robust to these issues.CONCLUSION: The UNAIDS 90-90-90 care continuum may lead to incorrect inference when used to evaluate health systems performance. We recommend that evaluation of HIV care delivery include longitudinal care continuum metrics wherever possible.
View details for DOI 10.1097/QAD.0000000000002502
View details for PubMedID 32044844
The worldwide clinical trial research response to the COVID-19 pandemic - the first 100 days.
2020; 9: 1193
Background: Never before have clinical trials drawn as much public attention as those testing interventions for COVID-19. We aimed to describe the worldwide COVID-19 clinical research response and its evolution over the first 100 days of the pandemic. Methods: Descriptive analysis of planned, ongoing or completed trials by April 9, 2020 testing any intervention to treat or prevent COVID-19, systematically identified in trial registries, preprint servers, and literature databases. A survey was conducted of all trials to assess their recruitment status up to July 6, 2020. Results: Most of the 689 trials (overall target sample size 396,366) were small (median sample size 120; interquartile range [IQR] 60-300) but randomized (75.8%; n=522) and were often conducted in China (51.1%; n=352) or the USA (11%; n=76). 525 trials (76.2%) planned to include 155,571 hospitalized patients, and 25 (3.6%) planned to include 96,821 health-care workers. Treatments were evaluated in 607 trials (88.1%), frequently antivirals (n=144) or antimalarials (n=112); 78 trials (11.3%) focused on prevention, including 14 vaccine trials. No trial investigated social distancing. Interventions tested in 11 trials with >5,000 participants were also tested in 169 smaller trials (median sample size 273; IQR 90-700). Hydroxychloroquine alone was investigated in 110 trials. While 414 trials (60.0%) expected completion in 2020, only 35 trials (4.1%; 3,071 participants) were completed by July 6. Of 112 trials with detailed recruitment information, 55 had recruited <20% of the targeted sample; 27 between 20-50%; and 30 over 50% (median 14.8% [IQR 2.0-62.0%]). Conclusions: The size and speed of the COVID-19 clinical trials agenda is unprecedented. However, most trials were small investigating a small fraction of treatment options. The feasibility of this research agenda is questionable, and many trials may end in futility, wasting research resources. Much better coordination is needed to respond to global health threats.
View details for DOI 10.12688/f1000research.26707.1
View details for PubMedID 33082937
- Improving the Validity of Mathematical Models for HIV Elimination by Incorporating Empirical Estimates of Progression Through the HIV Treatment Cascade JAIDS-JOURNAL OF ACQUIRED IMMUNE DEFICIENCY SYNDROMES 2018; 79 (5): 596?604
Improving the validity of mathematical models for HIV elimination by incorporating empirical estimates of progression through the HIV treatment cascade.
Journal of acquired immune deficiency syndromes (1999)
BACKGROUND: Optimism regarding prospects for eliminating HIV by expanding antiretroviral treatment has been emboldened in part by projections from several mathematical modeling studies. Drawing from a detailed empirical assessment of rates of progression through the entire HIV care cascade, we quantify for the first time the extent to which models may overestimate health benefits from policy changes when they fail to incorporate a realistic understanding of the cascade.SETTING: Rural KwaZulu-Natal, South Africa METHODS:: We estimated rates of progression through stages of the HIV treatment cascade using data from a longitudinal population-based HIV surveillance system in rural KwaZulu-Natal. Incorporating empirical estimates in a mathematical model of HIV progression, infection transmission, and care, we estimated life expectancy and secondary infections averted under a range of treatment scale-up scenarios reflecting expanding treatment eligibility thresholds. We compared the results to those implied by the conventional assumptions that have been commonly adopted by existing models.RESULTS: Survival gains from expanding the treatment eligibility threshold from CD4 350 to 500 cells/muL and from 500 cells/muL to treating everyone irrespective of their CD4 count may be overestimated by 3.60 and 3.79 times in models that fail to capture realities of the care cascade. HIV infections averted from raising the threshold from CD4 200 to 350, 350 to 500, and 500 cells/muL to treating everyone may be overestimated by 1.10, 2.65, and 1.18 times.CONCLUSION: Models using conventional assumptions about cascade progression may substantially overestimate health benefits. As implementation of treatment scale-up proceeds, it is important to assess the effects of required scale-up efforts in a way that incorporates empirical realities of how people move through the HIV cascade.This is an open-access article distributed under the terms of the Creative Commons Attribution-Non Commercial License 4.0 (CCBY-NC) , where it is permissible to download, share, remix, transform, and buildup the work provided it is properly cited. The work cannot be used commercially without permission from the journal.
View details for PubMedID 30272631