If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Send correspondence and reprint requests to Nicholas J. Ainsworth, M.D., Centre for Addiction and Mental Health, 6th Floor, 80 Workman Way, Toronto, ON M6J 1H4, Canada.
Centre for Addiction and Mental Health, Toronto, ON, CanadaDepartment of Psychiatry, Temerty Faculty of Medicine, University of Toronto, Toronto ON, Canada
Centre for Addiction and Mental Health, Toronto, ON, CanadaDepartment of Psychiatry, Temerty Faculty of Medicine, University of Toronto, Toronto ON, Canada
Department of Psychiatry, Temerty Faculty of Medicine, University of Toronto, Toronto ON, CanadaToronto General Hospital Research Institute, University Health Network, Toronto, ON, Canada
Centre for Addiction and Mental Health, Toronto, ON, CanadaDepartment of Psychiatry, Temerty Faculty of Medicine, University of Toronto, Toronto ON, Canada
What is the primary question addressed in this study?
This study evaluated online recruitment as a strategy to increase enrollment in a clinical trial for late-life depression during COVID-19, comparing its effectiveness and outcomes to traditional recruitment through provider referrals.
•
What is the main finding of this study?
This study found that online recruitment increased enrollment, though the rate of enrollment (yield) from self-referrals was lower than from provider referrals. Online self-referrals increased during times of high public health stringency and decreased during times of lower stringency, and vice-versa for provider referrals.
•
What is the meaning of the finding?
Online recruitment is an effective strategy for increasing clinical trial enrolment and may be particularly useful during times of public health restriction, but the lower yield of this approach suggests that cost-effectiveness is a potential concern.
Abstract
Objective
To evaluate the effectiveness of online recruitment for a clinical trial of pharmacotherapy for late-life depression during COVID-19.
Methods
The authors calculated the yield, defined as recruitment leading to randomization (enrollment), from provider referrals versus Facebook self-referrals; compared characteristics and drop-out rates of participants from each source; and analyzed correlations between stringency of public health restrictions and referrals from each source over time.
Results
Provider referrals had a significantly higher yield (10 of 33 referrals; 30.3%) versus Facebook self-referrals (14 of 323; 4.3%) (p <0.00001). Participants self-referred from Facebook had significantly more education; otherwise, both groups had similar characteristics and drop-out rates. While public health stringency was negatively correlated with provider referrals (ρ = −0.32) and positively correlated with Facebook self-referrals (ρ = 0.39), neither association reached statistical significance.
Conclusion
Online recruitment may improve access to clinical research for older depressed adults. Future studies should evaluate cost-effectiveness and potential barriers such as computer literacy.
The effectiveness of social media (Facebook) compared with more traditional advertising methods for recruiting eligible participants to health research studies: a randomized, controlled clinical trial.
with Facebook the most common recruitment platform. Generally, these studies reported success in recruiting eligible participants. Limited available data on cost-effectiveness show a high expenditure per enrolled participant, and with a possible increased cost as the trial progresses.
The effectiveness of social media (Facebook) compared with more traditional advertising methods for recruiting eligible participants to health research studies: a randomized, controlled clinical trial.
To date, reports have mostly used data from observational or survey-based research, for which online consent procedures were already established before the pandemic. There is limited evidence for the effectiveness of online clinical trial recruitment for psychiatric disorders, especially during COVID-19. One study of PTSD reported increased enrollment following the adoption of virtual procedures, including recruitment and consenting.
There is no evidence, to our knowledge, for the effectiveness of online clinical trial recruitment efforts during COVID-19 in older adults, who experience more barriers to technology use but were particularly susceptible to health risks of in-person contact during the pandemic.
Therefore, we investigated the impact of using Facebook on enrollment during COVID-19 in a clinical trial of pharmacotherapy for late-life depression (Lenze et al., under review). Our objectives were to: 1) describe the overall yield of provider referrals versus Facebook advertising for enrolling participants; 2) compare baseline characteristics and drop-out rates of enrolled participants who were referred by providers versus those who self-referred via Facebook; and 3) evaluate the association between COVID-19 public health measures and the performance of traditional versus online recruitment. We hypothesized that the yield from traditional referrals would be higher than the yield of Facebook self-referrals, and that increased stringency of public health measures would be associated with decrease in provider referral volume but increase in Facebook self-referral volume.
METHODS
Study Background
The Optimizing Outcomes of Treatment-Resistant Depression in Older Adults (OPTIMUM) study was a randomized comparative effectiveness trial that compared several pharmacotherapy strategies for management of treatment-resistant late-life depression (Lenze et al., under review). OPTIMUM was conducted at four sites in the United States and one site in Canada (Toronto), following approval by institutional review boards at each site. Recruitment in Toronto started in March 2017 and ran until the end of January 2022.
Recruitment Procedures
Throughout the duration of the study, participants were recruited via referrals from their PCP or psychiatrists. In the Spring 2020, the onset of the pandemic led to a marked decrease in the number of referrals and enrollments. In May 2020, the Toronto study team initiated a Facebook advertising campaign aimed at older adults (aged 60+) in the Greater Toronto Area, which ran until the end of January 2022 when the study concluded (see Image, Supplemental Digital Content 1, which shows the sample ads). We set a maximum spend of $20–$40 CAD per day, increasing the budget when our recruitment numbers were low to identify more potential participants. After reaching the selected daily maximum, Facebook automatically stopped presenting the ad. Interested individuals who clicked on the ad were invited to complete a confidential online self-referral form, with submitted forms reviewed by the study team.
Individuals referred to the study from either source were contacted for a phone prescreen. Participants who appeared eligible were approached for consent, and then invited to complete a baseline assessment consisting of questionnaires on mood, medical and medication history for a final confirmation of their eligibility. Eligible participants were randomized to a study intervention and are considered “enrollments” in our analysis.
Analysis
We calculated the number of enrollments from each referral source between May 2020 and January 2022. Statistical analyses using Fisher's exact test compared i) the yield for the two referral sources, defined as the number of participants who enrolled divided by the number of individuals who were referred, and ii) the rate of enrolment after being assessed as eligible. Student's t test and Fisher's exact test, as appropriate, compared sociodemographic and baseline clinical characteristics of participants recruited from each referral source, as well as rates of drop-out prior to completion of the Week 10 visit (which was the main efficacy endpoint of the trial). Finally, we analyzed fluctuations in referrals from both sources depending on the stringency of public health restrictions over time, using the mean Stringency Index for Ontario provided by the Bank of Canada.
Spearman's rank correlation was used to examine the association between number of referrals using each method and the mean Stringency Index for each month during the recruitment period.
RESULTS
The yield of each recruitment approach (i.e., enrolled participants divided by referrals) was 10 of 33 (30.3%) for traditional referrals and 14 of 323 (4.3%) for Facebook self-referrals (see Figures, Supplemental Digital Content 2 and 3, which show referral flow for traditional and Facebook referrals respectively). These two yields differed significantly (Fisher's exact test, p <0.00001). The rate of enrolment after being assessed as eligible was higher for participants recruited from providers (10/14; 71.4%) than from Facebook (14/27; 51.9%) but this difference was not significant (Fisher's exact test, p = 0.32).
The total spent on Facebook advertisements during the study period was CAD$16,778, for an average of CAD$1,198 corresponding to US$956, spent per successful enrollment.
Table 1 summarizes the demographic and clinical characteristics of the 10 participants recruited from provider referrals and the 14 participants recruited from Facebook self-referrals. Participants enrolled following Facebook self-referral had significantly more years of formal education than those recruited from provider referrals, but otherwise characteristics of both groups did not significantly differ. The rate of drop-outs before completion of the Week 10 follow-up visit did not differ between participants recruited from provider referrals (2/10, 20%) versus Facebook self-referrals (2/12, 16.7%) (Fisher's exact test, p = 1).
TABLE 1Summary of Sociodemographic and Clinical Characteristics of Participants Referred by Providers Versus Those Self-Referred Via Facebook
Figure 1 displays the number of monthly referrals and the stringency of public health measures over time. The Stringency Index was negatively correlated with the monthly number of referrals by health providers (Spearman's ρ = −0.32) and positively correlated with the monthly number of Facebook self-referrals (Spearman's ρ = 0.39), though neither correlation reached statistical significance (p = 0.15 and p = 0.08, respectively).
FIGURE 1Stringency index versus referral volume by source per month. Stringency Index—mean monthly value of the Bank of Canada's Stringency Index for Ontario, an estimate of public health stringency.
We report outcomes of an online recruitment initiative for a pharmacotherapy trial for treatment-resistant late-life depression during the COVID-19 pandemic. Overall, it was successful, leading to a larger absolute number of enrollments from online self-referrals than from traditional referral sources. However, as we had hypothesized, the yield of self-referrals was significantly lower than the yield of provider referrals. This may reflect the relatively narrow trial eligibility criteria (i.e., the requirement to be treatment resistant as evidenced by having failed to remit after at least two antidepressant trials of adequate dosage and duration).
Participants from each source did not differ demographically in most aspects. The similar age between referral groups suggests that age does not pose a fundamental barrier to online recruitment in older depressed adults. However, participants who self-referred from Facebook had a higher level of formal education, a difference which may reflect the association between computer literacy and education level.
Drop-out rates were similar after successful enrollment from either referral source, in contrast to earlier data suggesting a higher drop-out rate among participants recruited online.
There was an apparent relationship between COVID-19 public health restrictions and the rate of referrals from each referral source: providers’ referrals increased during times of low stringency, and self-referrals increased during times of high stringency. However, these correlations were not statistically significant, possibly due to the small sample size. Additional data are needed to confirm that online recruitment can increase self-referrals to clinical studies during periods of public health restriction.
While we have established feasibility of online recruitment for a clinical trial enrolling older adults with mental illness, our data also illustrate some challenges to this approach, particularly our finding of a low yield of Facebook self-referrals. Published data suggest that this pattern of high-engagement but low-yield is also observed with all other “broadcast” recruitment methods—for example, online, radio, or print advertisements, which produce higher numbers of referrals and lower yields than traditional referrals from clinicians.
Researchers designing similar trials must anticipate needing a high volume of self-referrals to achieve target enrollment. As done in our trial, using provider and online referrals concurrently can be an effective way of optimizing recruitment. Another consideration is those with lower education, for whom computer literacy is a potential barrier.
The low yield of online self-referrals also raises concerns about the cost-effectiveness of this approach. In our small study, we estimate that enrolling one participant via online referrals had a direct cost of $956 USD per enrolled participant; this cost is only for the Facebook ads and does not include the cost of research staff time needed to contact and screen the large number of people who self-refer in response to these ads. One potential strategy to improve the cost-effectiveness of online recruitment is to use targeted advertisement. In a recent clinical trial in atrial fibrillation, the investigators employed an iterative approach to recruitment, whereby the yield of each recruitment phase was evaluated and further iterations was tailored to the target population based on early patterns of response.
Digital recruitment and enrollment in a remote nationwide trial of screening for undiagnosed atrial fibrillation: Lessons from the randomized, controlled mSToPS trial.
More sophisticated approaches, for example using profiling data from Internet users, would need to balance the benefits of research access against potential privacy and ethics concerns.
The primary limitation of our study was the small size and lack of diversity of our sample, which limited the power of our statistical analyses, potentially leading to type II error. However, our results can be considered hypothesis-generating. Additionally, while we estimated the added direct costs of online recruitment (i.e., Facebook advertising fees), we did not have sufficiently precise data to calculate labor costs, specifically those associated with screening Facebook self-referrals or outreach to clinicians and screening clinicians’ referrals.
CONCLUSION
The COVID-19 pandemic has forced a re-think of delivery of healthcare services, including clinical research. Our study shows how online recruitment can mitigate participation barriers during a public health crisis, which is particularly important in vulnerable populations such as older adults and those with mental illness.
Further research should confirm effectiveness and assess cost-effectiveness in larger participant samples, and identify predictors of successful enrollment from online sources to enable improved targeting of resources.
AUTHORS’ CONTRIBUTIONS
Dr. Ainsworth was primarily responsible for conducting the analysis and writing the manuscript. Dr. Ainsworth, Ms. Wright, and Dr. Mulsant contributed to the study concept and design. Ms. Wright, Tereschenko, and Perivolaris were responsible for gathering data and maintaining the dataset used in this analysis. Drs. Blumberger, Flint, Lenze, and Mulsant contributed to the design and implementation of the OPTIMUM clinical trial and contributed to the interpretation and reporting of the present analysis.
PREVIOUS PRESENTATIONS
The data have not previously been presented orally or by poster at scientific meetings.
DISCLOSURES
Funding for the OPTIMUM trial includes grant support from the Patient-Centred Outcomes Research Institute (PCORI) TRD-1511-33321 and the National Institutes of Health (NIH) 3R01MH114980-03S1.
Dr. Ainsworth receives salary support from the University of British Columbia Clinician Investigator Program. Dr. Blumberger receives research support from the Canadian Institutes of Health Research (CIHR), National Institutes of Health – U.S. (NIH), Brain Canada Foundation and the Temerty Family through the CAMH Foundation and the Campbell Family Research Institute. He received research support and in-kind equipment support for an investigator-initiated study from Brainsway Ltd. and he was the site principal investigator for three sponsor-initiated studies for Brainsway Ltd. He received in-kind equipment support from Magventure for investigator-initiated studies. He received medication supplies for an investigator-initiated trial from Indivior. He has participated in an advisory board for Janssen. He has participated in an advisory board for Welcony Inc. Dr. Flint has received grant support from the U.S. National Institutes of Health, Patient-Centred Outcomes Research Institute, Canadian Institutes of Health Research, Brain Canada, Ontario Brain Institute, Alzheimer's Association, AGE-WELL, and the Canadian Foundation for Healthcare Improvement. Dr. Lenze receives grant support from COVID early treatment fund, Fast Grants, PCORI, Janssen; consulting fees from Merck, IngenioRx, Boeringer-Ingelheim, Pritikin ICR, Prodeo; patent pending for sigma 1 receptor agonists for COVID-19. Dr. Mulsant holds and receives support from the Labatt Family Chair in Biology of Depression in Late-Life Adults at the University of Toronto. He currently receives or has received with the past 3 years research support from Brain Canada, the Canadian Institutes of Health Research, the CAMH Foundation, the Patient-Centered Outcomes Research Institute (PCORI), the U.S. National Institute of Health (NIH), Capital Solution Design LLC (software used in a study founded by CAMH Foundation), and HAPPYneuron (software used in a study founded by Brain Canada). Within the past 3 years, he has also been an unpaid consultant to Myriad Neuroscience. Ms. Wright, Tereschenko, and Perivolaris have no disclosures.
The effectiveness of social media (Facebook) compared with more traditional advertising methods for recruiting eligible participants to health research studies: a randomized, controlled clinical trial.
Digital recruitment and enrollment in a remote nationwide trial of screening for undiagnosed atrial fibrillation: Lessons from the randomized, controlled mSToPS trial.
Significant difference between groups at p <0.05.
00154-9&id=gr1.jpg){kind=link}