by Jacqueline Lutz, PhD; Abhishek Pratap, PhD; Eric J. Lenze, MD; Durga Bestha, MD; Jessica M. Lipschitz, PhD; Stella Karantzoulis, PhD; Uma Vaidyanathan, PhD; Jessica Robin, PhD; William Horan, PhD; Stephen Brannan, MD; Aurelia Mittoux, PhD; Michael C. Davis, MD, PhD; Shaheen E. Lakhan, MD, PhD, MEd, MS, FAAN;* and Richard Keefe, PhD*

Dr. Lutz was with Medical Office, Click Therapeutics, Inc. in New York, New York, at the time of writing; she is now with Biogen Digital Health in Cambridge, Massachusetts, and Boston University School of Medicine in Boston, Massachusetts. Dr. Pratap was with Center for Addiction & Mental Health in Toronto, Canada, at the time of writing; he is now with Boehringer Ingelheim in Ridgefield, Connecticut; King’s College London in London, United Kingdom; and Department of Biomedical Informatics and Medical Education, University of Washington in Seattle, Washington. Dr. Lenze is with Department of Psychiatry, Washington University School of Medicine in St. Louis, Missouri. Dr. Bestha is with Atrium Health in Charlotte, North Carolina. Dr. Lipschitz is with Brigham and Women’s Hospital in Boston, Massachusetts, and Harvard Medical School in Boston, Massachusetts. Dr. Karantzoulis is with IQVIA in New York, New York. Dr. Vaidyanathan was with Boehringer Ingelheim in Ridgefield, Connecticut, at the time of writing; she is now with Sublimus in Ridgefield, Connecticut. Dr. Robin is with Winterlight Labs, Inc. in Toronto, Canada. Dr. Horan was with WCG VeraSci in Durham, North Carolina, at the time of writing; he is now with Karuna Therapeutics in Boston, Massachusetts, and University of California in Los Angeles, California. Dr. Brannan is with Karuna Therapeutics in Boston, Massachusetts. Dr. Mittoux is with Lundbeck in Paris, France. Dr. Davis is with Usona Institute in Madison, Wisconsin. Dr. Lakhan is with Medical Office, Click Therapeutics, Inc. in New York, New York, and School of Neuroscience, Virginia Tech in Blacksburg, Virginia. Dr. Keefe is with Department of Psychiatry, Duke University Medical Center in Durham, North Carolina.

*Shared last authors.

Funding: No funding was provided for this article.

Disclosures: Dr. Lutz may have equity interests with Click Therapeutics, Inc. Dr. Lenze has previously consulted for Merck, Prodeo, Boehringer Ingelheim, Pritikin ICR, and IngenioRx; he has previously received funding from PCORI, the COVID-19 Early Treatment Fund, Mercatus/FastGrants, and Janssen; and he has a patent pending for sigma1 receptor agonists for COVID-19 treatment. Dr. Vaidyanathan was an employee of Boehringer Ingelheim at the time of writing. Dr. Robin is an employee of Winterlight Labs, Inc. and has equity interests with Winterlight Labs, Inc. Dr. Horan was an employee of WCG VeraSci at the time of writing, and is now an employee of Karuna Therapeutics. Dr. Brannan is an employee at Karuna Therapeutics. Dr. Lakhan is employed by Click Therapeutics, Inc. and has equity interests. Dr. Keefe serves as a paid consultant to WCG, Karuna, Merck, Sunovion, Biogen, and Boehringer Ingelheim. All other authors have no conflicts of interest relevant to the contents of this article.

Innov Clin Neurosci. 2023;20(7–9):40–46.

Abstract

Objective: Recruitment of a sufficiently large and representative patient sample and its retention during central nervous system (CNS) trials presents major challenges for study sponsors. Technological advances are reshaping clinical trial operations to meet these challenges, and the COVID-19 pandemic further accelerated this development.

Method of Research: The International Society for CNS Clinical Trials and Methodology (ISCTM; www.isctm.org) Innovative Technologies for CNS Trials Working Group surveyed the state of technological innovations for improved recruitment and retention and assessed their promises and pitfalls.

Results: Online advertisement and electronic patient registries can enhance recruitment, but challenges with sample representativeness, conversion rates from eligible prescreening to enrolled patients, data privacy and security, and patient identification remain hurdles for optimal use of these technologies. Electronic medical records (EMR) mining with artificial intelligence (AI)/machine learning (ML) methods is promising but awaits translation into trials. During the study treatment phase, technological innovations increasingly support participant retention, including adherence with the investigational treatment. Digital tools for adherence and retention support take many forms, including patient-centric communication channels between researchers and participants, real-time study reminders, and digital behavioral interventions to increase study compliance. However, such tools add technical complexities to trials, and their impact on the generalizability of results are largely unknown.

Conclusion: Overall, the group found a scarcity of systematic data directly assessing the impact of technological innovations on study recruitment and retention in CNS trials, even for strategies with already high adoption, such as online recruitment. Given the added complexity and costs associated with most technological innovations, such data is needed to fully harness technologies for CNS trials and drive further adoption.

Keywords: Clinical trials, recruitment, e-consent, technology, virtual trials, CNS

Timely recruitment and retention (i.e., keeping a representative study sample enrolled in a trial) of diverse and representative study populations presents major bottlenecks for conducting clinical trials, which drives up costs and delays new treatments that are urgently needed. The “right” patients not only have to fulfill trial inclusion and exclusion criteria, but they should also be representative and retained (i.e., adhere to the clinical intervention being studied and be able to comply with study procedures over the course of the trial). There is evidence that most clinical trials do not achieve their original recruitment targets and timelines.1–3 This results in early termination of trials,4 underpowered studies in which clinically relevant differences are missed,5 and/or extended trial lengths, delaying evaluation and roll-out of potentially new and effective interventions for patients.2 A position paper by the National Institutes of Health (NIH) and the National Science Foundation (NSF) concluded that “current methods for conducting clinical trials are not sustainable and will leave a chasm between the need for evidence to inform health and healthcare and the availability of that evidence.”6

Clinical trials in central nervous system disorders (CNS) face even further challenges in the evaluation of novel treatments. In contrast to other therapeutic areas, such as oncology or cardiovascular diseases, there are no large clinical research networks for CNS, making it more difficult to efficiently reach patients in their care setting. There are also challenges associated with CNS disorders themselves that are not relevant to other therapeutic areas. For example, patients with CNS disorders often present with a myriad of symptoms across motor-sensory, cognitive, emotional, and behavioral domains, all of which can have a significant impact on day-to-day functioning. This adds to the already challenging task of maintaining active participation in a traditionally conducted clinical trial, which tends to be burdensome for patients. Thus, the timely recruitment and retention of the right and representative patients is a challenge for CNS clinical trials.

Technological advances can help alleviate these challenges. Telehealth visits, remote assessments, online advertisement, and electronic informed consent (e-consent) have been reshaping clinical trial operations and management in fully remote or hybrid (i.e., partly remote, partly in person) clinical trials.7 The COVID-19 pandemic further accelerated this development.6,8,9

Do such technologies hold the promise for the efficient recruitment of a representative patient population and their retention in CNS trials? What are the potential pitfalls when using such technologies in CNS clinical trials? In this article, the International Society for CNS Clinical Trials and Methodology (ISCTM) Innovative Technologies for CNS Trials Working Group examined technological innovations to improve the recruitment and retention of a representative patient population, starting with a general overview of technologies across the different clinical trial stages. In subsequent sections, key technologies to support recruitment and retention, including adherence to study protocol and treatment, are reviewed in more detail, and advantages and pitfalls associated with these technologies are discussed, along with the working groups’ recommendations on how to best advance these technologies to enable more efficient CNS trial recruitment and retention.

General Overview of Technological Trial Innovations

With technology becoming part of peoples’ daily lives,10 a wide range of technology-based applications (apps) have been developed to enable and aid the design and execution of early- and late-stage CNS trials.11 Here, we provide an overview of these technologies, with a particular focus on addressing key challenges related to recruitment and retention of patients in CNS trials. As shown in Figure 1, technological applications can be broadly classified into four categories: 1) general technology (e.g., internet or video conferencing) that is used for recruitment and study visits; 2) healthcare systems, which provide data that may aid in matching the right patient to clinical trials and support recruitment; 3) clinical research technology, which assists in recruitment and retention throughout the study phases and enables e-consent that is particularly relevant for efficient enrollment; and 4) digital health technologies and applications that can assist with treatment adherence, motivation, and monitoring during the treatment study phase. The extent to which each technological approach seems adopted in CNS trials is color-coded as “well-established” (green), “beginning adoption” (tan), or “still significant hurdles” (red), based on the working group’s impressions of the literature and knowledge of use cases to date.

Technologies from Recruitment to Enrollment

Technology has been increasingly used to reach and recruit potentially eligible participants for clinical trials. This includes online recruitment, such as targeted online advertisement or web-based registries, technological advances related to mining electronic medical record (EMRs) for efficient trial recruitment, and e-consent technology. Here, we provide a summary of opportunities and challenges associated with these technologies (Table 1).

Online platforms. Several online recruitment strategies are available, from targeted online advertisements on social media and special interest or clinical websites, to web-based patient registries. Typically, to recruit the right participants from large and diverse populations, researchers post study advertisements and links on such platforms, guiding potentially interested participants to online study recruitment portals. Such approaches are promising in effectively recruiting diverse and harder-to-reach patients (e.g., patients in rural areas or patients who do not usually participate in clinical trials) and promise to reach trial recruitment goals faster and with lower costs. A recent meta-analysis of 23 clinical trials found that online recruitment was faster and more cost-effective, compared to offline recruitment.12 Social media platforms, such as Reddit or Facebook, are becoming common recruitment platforms for clinical trials.13

Large patient registries provide another online recruitment channel for clinical trials. For example, MindCrowd®, a collaboration between academic, nonprofit institutions and the Alzheimer’s Prevention Initiative (API), is currently collecting observational data and offering researchers the opportunity to promote their studies with registry patients.14 A similar initiative from the European Prevention of Alzheimer’s Dementia (EPAD) Consortium established the first pan-European registry of research participants across the dementia risk spectrum, enabling massive longitudinal observational studies and providing trial-ready samples (i.e., patients who meet eligibility criteria in the indication under study and can more efficiently be enrolled into interventional studies) for two intervention studies. Online trial-matching registries and linked remote studies have also been established in Parkinson’s disease.15,16

However, the effectiveness of online platforms for recruiting the right patients appears variable. A review suggested that it might be the best recruitment method for hard-to-reach populations and observational studies, but the overall success and effectiveness for interventional studies was more mixed.17 In the case of registries, attrition of registry patients is a known issue. In addition, Aysen et al18 reported challenges in recruiting participants into the API registry project and loss of participants to follow-up, and the EPAD initiative did not lead to actual clinical studies based on a trial-ready cohort within the funded period, mainly related to delays caused by the COVID-19 pandemic. Registries also require particularly sophisticated and secure technological database systems to enable data exchange between trial-ready cohorts and clinical studies.19

Whether the use of online platforms will be effective in recruiting the right patients for CNS clinical trials will depend on whether the patient population has access to and uses social media and the internet. There are still large racial/ethnic differences in who accesses the internet and how it is accessed.10 In addition, social media users might not be representative regarding sex, age, socioeconomic status, or education level20–22 for the target population that interests the sponsor, and the same issues have been reported for registry trials.19 For example, late-stage Alzheimer’s disease trials typically recruit older, cognitively impaired populations who are less likely to be active on any social media platforms.23 

Recruitment effectiveness also further depends on patient willingness to self-disclose information that can be targeted through advertisement (i.e., demographic, sociographic, or relevant health information).13 The openness of social media users to share such information might differ depending on the target population. Such differences can lead to the recruitment of imbalanced cohorts that are not representative of the patient population of interest.

Additional challenges relate to federally mandated requirements for the privacy and security of patients’ protected health information (PHI) when using social media for recruitment. Data exchanged through social media channels is often permanent and outside of the researcher’s control. Given that social media platforms are common targets of hackers, data breaches on social media platforms containing PHI may not be readily correctable, which may lead to civil penalties based on the Health Insurance Portability and Accountability Act (HIPAA).13

There are additional caveats to online recruitment in trials. While generally cost-effective and timely, online recruitment may yield lower conversion rates from recruitment to enrollment than offline recruitment, which might be related to offline screening often involving patients who already expressed interest in participating or referrals who were prequalified for enrollment.12 Thus, while potentially faster, online recruitment can create novel pain points for sponsors, sites, and researchers, as an increasing number of ineligible individuals will have to be screened out.24 Furthermore, verifying patient identity requires novel approaches for trials conducted in a virtual format. Remote identity validation and identity proofing services based on credit history25 or using social media profiles or other online accessible personal demographic information are potential approaches to verify participant identity during screening.

EMR mining for clinical trial recruitment. A large volume of real-world health data is gathered at the point of care by healthcare systems and recorded in several databases, including EMRs, claims, billing records, digital images (computed tomography [CT], magnetic resonance imaging [MRI], etc.), and clinical trial data. These real-world data have the potential to effectively identify potential study participants through matching with the trial inclusion/exclusion conditions.

To date, clinical trial-matching systems based on EMR mining have mainly been used in cancer trials as a clinician support tool. Systems generally demonstrated high accuracy and time savings, compared to manual review.26–28 Although IBM’s cognitive computing/artificial intelligence (AI) for healthcare program received criticism for underdelivering on promises of advancing medical diagnoses and treatment-recommendations,29 the IBM Watson for Clinical Trial Matching (CTM) reported timely and effective assessment of site feasibility and identification of cohorts of potentially eligible patients for oncology trials.30 In cancer research that defines more and narrower gene-level cancer indications for targeted treatments, the need for efficient recruitment of narrow patient populations drove innovation in this area.31 Thus, technologies such as AI for patient screening have matured to the level of assisting human decision-makers.32 CNS trials have not seen the same level of innovation in this area to date. Nevertheless, in CNS trials, AI-assisted trial eligibility determinations could yield sizable improvements over standard practices in several aspects of the patient recruitment process.33 When accessible, real-world data sources, such as EMRs, offer the potential for identifying patients with conditions of interest for clinical trials and provide data-driven estimates for recruitment feasibility;34,35 it seems a matter of time until such systems will be more widely used for CNS trials.

However, there are several challenges in utilizing EMR data for identifying and recruiting the right patients into clinical trials. Among the challenges are data access governance, or regulations on who can and how to access EMR data, patient privacy, and security, all of which are still evolving, thus limiting the use of these systems for research.32 Other challenges are related to a lack of data standardization and to data not often being centralized or integrated within and across care-providing institutions.36 A specific example is the lack of standardized terminology or symptoms beyond the Diagnostic and Statistical Manual of Mental Disorders (DSM)/International Classification of Diseases (ICD) in psychiatry, where simple terms like “mood” and “affect” lack standardized definitions across databases. Thus, unlike fields such as cancer or cardiology that have standardized terms describing disease mechanisms, EMR data in psychiatry contain patient and clinician symptom reporting, and sometimes even lay person language, which makes integration across EMR databases even more difficult. Additionally, there are challenges related to the representativeness of EMR data; recent research findings show that real-world care data in EMRs might have biases related to the representation of the target population (i.e., selection bias related to who is able to access the care system due to geospatial and socioeconomic factors).37–39

E-consent. Consent procedures are traditionally conducted during site visits. In addition to presenting a burden to the participant or limiting enrollment to patients who live close to clinical sites, manual tracking and management of participant screening procedures is cumbersome and can lead to errors or duplicate patient data. Clinical trial technology, such as patient database systems integrated with e-consent systems,40 address these pain points and are increasingly implemented in clinical trials.41

Beyond the efficient gathering of e-consent, such systems may offer additional benefits for study enrollment. For example, they may use animations or quizzes to provide study information or offer opportunities to discuss the study with investigators via chat functions. By adapting to different cultural needs and levels of health literacy,42 e-consent could help increase engagement with and time spent reviewing the informed consent,43 ultimately improving patient understanding of the clinical trial. Patient-centered use of technology could even include chatbots during enrollment to assess patient eligibility using simple language or support trial understanding.44 Such personalized, on-demand, patient-centric technologies might enable enrollment of more diverse patient samples. As clinical trials become increasingly automated and virtual, we might see an increase in patient-centered technology replacing human factors, increasing engagement and leading to more diverse samples. At the same time, the use of more patient-centered technologies in clinical trials might have effects on the placebo response, stemming, for example, from beliefs in technology or perceptions of being more connected to clinical trial staff and researchers through technology.45

Working group recommendations for technologies in the recruitment stage. In general, the working group recommends the increased use of online recruitment strategies to achieve recruitment goals more quickly. At the same time, there is still limited data to fully understand comparative cost and time efficiencies of specific online recruitment channels and strategies across different CNS indications. More systematic data should be gathered and shared to enable study sponsors to make informed decisions for the specific indications under study.

There are also limited data available assessing whether specific online recruitment campaigns lead to more or less representative samples. It is likely that the answer will depend on the target population and the channels being used. The group considers recruitment vendors a valuable option to help balance nonrepresentative samples resulting from online recruitment.

Regarding data privacy and security, the group recommends that clinical trial technology supporting patient recruitment and identity verification should employ the highest standards to protect subject rights, enable confidentiality, and protect PHI by complying with provisions such as HIPAA in the United States, General Data Protection Regulation (GDPR) in the European Union (EU), and California Consumer Privacy Act (CCPA) in California. Systems should, for example, comply with data security standards, including two-factor authentication, or purge functions (i.e., the removal of obsolete data) in electronic databases. Furthermore, distributed ledgers and decentralized databases might mitigate the risk of database breaches. Protecting data privacy is particularly important when using digital tools that could be, or could be perceived to be, intrusive or that create risk of exposing personal identifiable data to bad actors during a clinical trial, such as with the use of social media or mobile apps.46 Data and safety monitoring plans should include potential risks to patient data privacy due to technology updates, server hacking, data sharing, and others.

Finally, e-consent is a patient-friendly tool that should be more widely accepted, used, and optimized to enable a diverse patient population and broad health literacy spectrum to understand the risks and benefits of participating in clinical studies; this includes adequately informing participants of data privacy risks related to the use of digital technologies for collecting personal, health-related information in a clinical trial.

Technologies to Support Patient Retention and Adherence

Considering the discussed challenges for patient recruitment, retention is another area of vital importance to study sponsors.47 In addition, enrolled participants should also adhere to study procedures and treatment to allow for the assessment of novel CNS treatments. Innovative technologies offer ways to potentially improve study retention, including adherence monitoring and support (Table 2).

Adherence monitoring and support. One potential way to improve overall trial retention and adherence is to use technology to track and improve levels of adherence or exclude nonadherent enrolled patients using trial run-in periods.

In digital therapeutics trials (i.e., trials assessing software-based therapies), adherence with the digital treatment can be directly assessed from the interaction with the software device. Trials may thus define target adherence rates during a run-in period and only randomize adherent participants. This principle is being implemented in a fully remote trial of a digital therapeutic app for patients with depression by Otsuka and Click Therapeutics at the time of this manuscript submission, and as such, no data on the effectiveness of this approach is available yet.48

Adherence monitoring for pharmaceutical trials is more challenging, as, in contrast to digital therapies, pharmaceutical therapies are not software-based. Technology has been used in various forms to support adherence monitoring, including digitally assessing self-reported adherence using electronic diaries,49 utilizing interactive voice response (IVR) calls to contact participants after dosing windows, measuring medication events using memory chips embedded in bottle caps (MEMS), or directly assessing medication adherence through ingestible sensors or breath recording.50,51 Most of these technologies either present additional patient burden or have not been used in many trials to date.52 A fully remote, phone-based technology with relatively low patient burden is an AI-based solution that confirms medication ingestion from camera-based facial recognition and motion-sensing technology (AiCure).53

A caveat to using technological tools for adherence monitoring is that they can impact the representativeness of a sample. Run-in phases to exclude a specific category of patients might result in clinically less applicable results.54 The size of this effect is hard to assess, as run-in period data are often not fully reported.55 Also, it should be noted that, while run-in periods are a common and accepted practice, excluding nonadherent patients after randomization raises issues with respect to the intent-to-treat principle. Beyond excluding patients with low adherence, treatment monitoring, especially if combined with reminders during a clinical study, might increase overall treatment adherence of enrolled participants.56

Retention support. Use of technology is not limited to monitoring adherence and identifying and excluding nonadherent patients; it can also assist with the retention of participants over time. Study websites or apps may deliver trial reminders or share updates about trial progress.42 Clinical trial technology may also directly deliver behavioral and educational interventions to enhance adherence to interventions,57,58 or they may offer additional channels and platforms for communicating with study staff. The use of study apps or similar technology may even be used to share patient data after the study, which could be a major draw for patients to enroll. Technology can also enable more active engagement, including the engagement of more diverse participants, following a community-based, participatory research model.59,60 However, the clinical relevance of trials with substantial technical adherence support tools will require further investigation.

Working group recommendations for technologies for patient retention. The working group recommends that studies should aim for a high level of retention and adherence and should explore digital health technologies, such as apps, to track and monitor study participants and their adherence to interventions during the trial. Thus, the use of study apps or the combination of digital therapeutics and pharmacotherapy have the potential to support adherence, enhance retention, and serve the participants’ information and communication needs in a synergistic manner.

At the same time, there are unknowns about the effects of increased digital support during CNS trials, for example, on the placebo response or effects on sample representativeness. Furthermore, using technology enabled run-in periods could affect sample representativeness. Such effects should be studied systematically, as new technologies are being adopted more often. To assess the generalizability of results, the group recommends that study reports should include the number and characteristics of eligible patients who were not randomized to understand how technologies or technology-supported run-in periods might affect the representativeness and retention of a sample.

Conclusion

The ISCTM is a multidisciplinary, independent organization devoted to promoting advances that address strategic clinical, regulatory, methodological, and policy challenges that arise in the development and use of CNS therapeutic agents. The ISCTM Innovative Technologies for CNS Trials Working Group sought input and expertise from a broad range of individuals spanning academia, industry, and regulatory agencies to review the status of the current technology and form recommendations and conclusions. 

The working group identified a variety of technological innovations that are available for CNS trial sponsors to recruit and retain a representative patient sample. Currently, however, there is a lack of systematic and large-scale data on the use of such technologies and their effects on trial costs and efficiencies for specific CNS indications. Until such data are more widely available, technologies that are more complex to implement or that pose challenges around data access and governance will likely see slower adoption. For example, EMR mining for trial recruitment has great potential for precision CNS trials, but the technology awaits adoption in CNS trials due to challenges with data quality, standards, and access regulations. It also remains to be seen whether novel technological innovations will enable more diverse and representative study cohorts (e.g., through more fully remote or hybrid studies) or pose additional challenges to representativeness (e.g., due to the digital divide). Further unknowns relate to how novel patient-centric technologies affect trial data and clinical meaningfulness of results. For example, the impact of such technologies on the placebo response is currently unknown. In the future, and as more systematic data become available, the group anticipates that technological innovations will be further adopted and optimized to lessen the burden of trial recruitment and retention.

Acknowledgments

The authors would like to thank Andrew Potter, PhD (US Food and Drug Administration), for their contributions in the process of developing this manuscript.

References

  1. McDonald AM, Knight RC, Campbell MK, et al. What influences recruitment to randomised controlled trials? A review of trials funded by two UK funding agencies. Trials. 2006;7(1):1–8.
  2. Treweek S, Lockhart P, Pitkethly M, et al. Methods to improve recruitment to randomised controlled trials: Cochrane systematic review and meta-analysis. BMJ Open. 2013;3(2):e002360.
  3. Walters SJ, Bonacho I, Henriques-Cadby A, et al. Recruitment and retention of participants in randomised controlled trials: a review of trials funded and published by the United Kingdom Health Technology Assessment Programme. BMJ Open. 2017;7(3):e015276.
  4. Williams RJ, Tse T, DiPiazza K, Zarin DA. Terminated trials in the ClinicalTrials.gov results database: evaluation of availability of primary outcome data and reasons for termination. PLoS One. 2015;10(5):e0127242.
  5. Altman DG. Statistics and ethics in medical research: III How large a sample? Br Med J. 1980;281(6251):1336–1338.
  6. Inan OT, Tenaerts P, Prindiville SA, et al. Digitizing clinical trials. NPJ Digit Med. 2020;3:101.
  7. Hirsch IB, Martinez J, Dorsey ER, et al. Incorporating site-less clinical trials Into drug development: a framework for action. Clin Ther. 2017;39(5):1064–1076.
  8. Gaba P, Bhatt DL. The COVID-19 pandemic: a catalyst to improve clinical trials. Nat Rev Cardiol. 2020;17(11):673–675.
  9. McDermott MM, Newman AB. Remote research and clinical trial integrity during and after the coronavirus pandemic. JAMA. 2021;325(19):1935–1936.
  10. Atske S, Perrin A. Home broadband adoption, computer ownership vary by race, ethnicity in the US. Pew Research Center. 16 Jul 2021. https://www.pewresearch.org/fact-tank/2021/07/16/home-broadband-adoption-computer-ownership-vary-by-race-ethnicity-in-the-u-s/. Accessed 27 Jun 2023.
  11. Mantua V, Arango C, Balabanov P, Butlen-Ducuing F. Digital health technologies in clinical trials for central nervous system drugs: an EU regulatory perspective. Nat Rev Drug Discov. 2020;20(2):83–84.
  12. Brøgger-Mikkelsen M, Ali Z, Zibert JR, et al. Online patient recruitment in clinical trials: systematic review and meta-analysis. J Med Internet Res. 2020;22(11):e22179.
  13. Sanchez C, Grzenda A, Varias A, et al. Social media recruitment for mental health research: a systematic review. Compr Psychiatry. 2020;103:152197.
  14. TGen. MindCrowd: leveraging genetics to advance Alzheimer’s research. https://mindcrowd.org/. Accessed 27 Jun 2023.
  15. Dorsey ER, Wagner JD, Bull MT, et al. Feasibility of virtual research visits in Fox Trial Finder. J Parkinsons Dis. 2015;5(3):505–515.
  16. The Michael J. Fox Foundation for Parkinson’s Research. Fox Insight. https://www.michaeljfox.org/fox-insight. Accessed 5 May 2022.
  17. Topolovec-Vranic J, Natarajan K. The use of social media in recruitment for medical research studies: a scoping review. J Med Internet Res. 2016;18(11):e286.
  18. Aisen PS, Sperling RA, Cummings J, et al. The Trial-Ready Cohort for Preclinical/Prodromal Alzheimer’s Disease (TRC-PAD) project: an overview. J Prev Alzheimer’s Dis. 2020;7(4):208–212.
  19. Aisen PS, Jimenez-Maggiora GA, Rafii MS, et al. Early-stage Alzheimer disease: getting trial-ready. Nat Rev Neurol. 2022;18(7):389–399.
  20. Whitaker C, Stevelink S, Fear N. The use of Facebook in recruiting participants for health research purposes: a systematic review. J Med Internet Res. 2017;19(8):e290.
  21. Guillory J, Kim A, Murphy J, et al. Comparing Twitter and online panels for survey recruitment of e-cigarette users and smokers. J Med Internet Res. 2016;18(11):e288.
  22. Thornton L, Batterham PJ, Fassnacht DB, et al. Recruiting for health, medical or psychosocial research using Facebook: systematic review. Internet Interv. 2016;4:72–81.
  23. Coto M, Lizano F, Mora S, Fuentes J. Social media and elderly people: research trends. In: Meiselwitz G, ed. Social Computing and Social Media. Applications and Analytics. Springer; 2017:65–81.
  24. Lattie EG, Kaiser SM, Alam N, et al. A practical do-it-yourself recruitment framework for concurrent eHealth clinical trials: identification of efficient and cost-effective methods for decision making (part 2). J Med Internet Res. 2018;20(11):e11050.
  25. Glazer J V., MacDonnell K, Frederick C, et al. Liar! Liar! Identifying eligibility fraud by applicants in digital health research. Internet Interv. 2021;25:100401.
  26. Alexander M, Solomon B, Ball DL, et al. Evaluation of an artificial intelligence clinical trial matching system in Australian lung cancer patients. JAMIA Open. 2020;3(2):209–215.
  27. Beck JT, Rammage M, Jackson GP, et al. Artificial intelligence tool for optimizing eligibility screening for clinical trials in a large community cancer center. JCO Clin Cancer Informatics. 2020;4(4):50–59.
  28. Ho TP, Helgeson JM, Andring AL, et al.  Implementation of cognitive computing to match clinical trials in gynecologic cancers: a single-institution experience [abstract]. In: Proceedings of the AACR Special Conference on Advancing Precision Medicine Drug Development: Incorporation of Real-World Data and Other Novel Strategies; Jan 9-12, 2020; San Diego, CA. American Association for Cancer Research. Clin Cancer Res. 2020;26(12 Suppl 1):Abstract nr 01.
  29. Strickland E. How IBM Watson overpromised and underdelivered on AI health care. IEEE Spectrum. 2 Apr 2019. https://spectrum.ieee.org/how-ibm-watson-overpromised-and-underdelivered-on-ai-health-care. Accessed 27 Jun 2023.
  30. Helgeson J, Rammage M, Urman A, et al. Clinical performance pilot using cognitive computing for clinical trial matching at Mayo Clinic. J Clin Oncol. 2018;36(15 Suppl):e18598.
  31. Content M, Now S, View VL. Pharma’s role in value-based care starts with value-based research. Statnews. 21 Apr 2022. https://www.statnews.com/2022/04/21/pharmas-role-in-value-based-care-starts-with-value-based-research/. Accessed 27 Jun 2023.
  32. Harrer S, Shah P, Antony B, Hu J. Artificial intelligence for clinical trial design. Trends Pharmacol Sci. 2019;40(8):577–591.
  33. Calaprice-Whitty D, Galil K, Salloum W, et al. Improving clinical trial participant prescreening with artificial intelligence (AI): a comparison of the results of AI-assisted vs standard methods in 3 oncology trials. Ther Innov Regul Sci. 2020;54(1):69–74.
  34. Evans SR, Paraoan D, Perlmutter J, et al. Real-world data for planning eligibility criteria and enhancing recruitment: recommendations from the Clinical Trials Transformation Initiative. Ther Innov Regul Sci. 2021;55(3):545–552.
  35. Marquis-Gravel G, Roe MT, et al. Technology-enabled clinical trials. Circulation. 2019;140(17):1426–1436.
  36. Behera RK, Bala PK, Dhir A. The emerging role of cognitive computing in healthcare: a systematic literature review. Int J Med Inform. 2019;129:154–166.
  37. Bastarache L, Brown JS, Cimino JJ, et al. Developing real-world evidence from real-world data: transforming raw data into analytical datasets. Learn Heal Syst. 2022;6(1):e10293.
  38. Haneuse S, Daniels M. A general framework for considering selection bias in EHR-based studies: what data are observed and why? EGEMS (Wash DC). 2016;4(1):1203.
  39. Agniel D, Kohane IS, Weber GM. Biases in electronic health record data due to processes within the healthcare system: retrospective observational study. BMJ. 2018;361:k1479.
  40. United States Food and Drug Administration. Use of electronic informed consent – questions and answers: guidance for Institutional Review Boards, investigators, and sponsors. 2016. https://www.fda.gov/media/116850/download. Accessed 27 Jun 2023.
  41. Skelton E, Drey N, Rutherford M, et al. Electronic consenting for conducting research remotely: a review of current practice and key recommendations for using e-consenting. Int J Med Inform. 2020;143:104271.
  42. Bower P, Brueton V, Gamble C, et al. Interventions to improve recruitment and retention in clinical trials: a survey and workshop to assess current practice and future priorities. Trials. 2014 Oct;15:399.
  43. Rowbotham MC, Astin J, Greene K, Cummings SR. Interactive informed consent: randomized comparison with paper consents. PLoS One. 2013;8(3):e58603.
  44. Chuan C-H, Morgan S. Creating and evaluating chatbots as eligibility assistants for clinical trials. ACM Trans Comput Healthc. 2021;2(1):1–19.
  45. Torous J, Firth J. The digital placebo effect: mobile mental health meets clinical psychiatry. Lancet Psychiatry. 2016;3(2):100–102.
  46. Pratap A, Allred R, Duffy J, et al. Contemporary views of research participant willingness to participate and share digital data in biomedical research. JAMA Netw Open. 2019;2(11):e1915717.
  47. Chaudhari N, Ravi R, Gogtay N, Thatte U. Recruitment and retention of the participants in clinical trials: challenges and solutions. Perspect Clin Res. 2020;11(2):64–69.
  48. Muoio D. Otsuka, Click Therapeutics kick off decentralized pivotal trial for depression digital therapeutics. MobiHealth News. 24 Feb 2021. https://www.mobihealthnews.com/news/otsuka-click-therapeutics-kick-decentralized-pivotal-trial-depression-digital-therapeutics. Accessed 27 Jun 2023.
  49. Farmer KC. Methods for measuring and monitoring medication regimen adherence in clinical trials and clinical practice. Clin Ther. 1999;21(6):1074–1090.
  50. Park LG, Howie-Esquivel J, Dracup K. Electronic measurement of medication adherence. West J Nurs Res. 2015;37(1):28–49.
  51. McCann DJ, Petry NM, Bresell A, et al. Medication nonadherence, “professional subjects,” and apparent placebo responders: overlapping challenges for medications development. J Clin Psychopharmacol. 2015;35(5):566–573.
  52. Shiovitz TM, Bain EE, McCann DJ, et al. Mitigating the effects of nonadherence in clinical trials. J Clin Pharmacol. 2016;56(9):1151–1164.
  53. Bain EE, Shafner L, Walling DP, et al. Use of a novel artificial intelligence platform on mobile devices to assess dosing compliance in a Phase 2 clinical trial in subjects with schizophrenia. JMIR Mhealth Uhealth. 2017;5(2):e18.
  54. Laursen DRT, Paludan-Müller AS, Hróbjartsson A. Randomized clinical trials with run-in periods: frequency, characteristics and reporting. Clin Epidemiol. 2019;11:169–184.
  55. Charlson ME, Horwitz RI. Applying results of randomised trials to clinical practice: impact of losses before randomisation. Br Med J. 1984;289(6454):1281–1284.
  56. Kini V, Michael Ho P. Interventions to improve medication adherence: a review. JAMA. 2018;320(23):2461–2473.
  57. Robiner WN. Enhancing adherence in clinical research. Contemp Clin Trials. 2005;26(1):59–77.
  58. Pérez-Jover V, Sala-González M, Guilabert M, Mira JJ. Mobile apps for increasing treatment adherence: systematic review. J Med Internet Res. 2019;21(6):e12505.
  59. Clark LT, Watkins L, Piña IL, et al. Increasing diversity in clinical trials: overcoming critical barriers. Curr Probl Cardiol. 2019;44(5):148–172.
  60. Sung NS, Crowley WF, Genel M, et al. Central challenges facing the national clinical research enterprise. JAMA. 2003;289(10):1278–1287.