ICNSl logo no-tagline copy

PEER REVIEWED, EVIDENCE-BASED INFORMATION FOR CLINICIANS AND RESEARCHERS IN NEUROSCIENCE

Special Challenges in Pediatric Drug Development: First of Three Sets of Expanded Proceedings from the ISCTM Autumn Conference on Pediatric Drug Development

by Philip D. Harvey, PhD; Joan Busner, PhD; Gahan Pandina, PhD; H. Gerry Taylor, PhD; Meg Grabb, PhD; and Joohi Jimenez-Shahed, MD 

Dr. Harvey is with Department of Psychiatry, University of Miami Miller School of Medicine in Miami, Florida. Dr. Busner is with Signant Health in Blue Bell, Pennsylvania and Department of Psychiatry, Virginia Commonwealth University School of Medicine in Richmond, Virginia. Dr. Pandina is with Janssen Pharmaceuticals in Titusville, New Jersey. Dr. Taylor is with Abigail Wexner Research Institute at Nationwide Children’s Hospital and The Ohio State University in Columbus, Ohio. Dr. Grabb is with the National Institute of Mental Health in Bethesda, Maryland. Dr. Jimenez-Shahed is with Icahn School of Medicine at Mount Sinai in New York, New York.

Funding: No funding was provided for this article.

Disclosures: Dr. Harvey has received consulting fees or travel reimbursements from Alkermes, Bio Excel, Boehringer Ingelheim, Karuna Pharma, Merck Pharma, Minerva Pharma, and Sunovion (DSP)Pharma in the past year. He receives royalties from the Brief Assessment of Cognition in Schizophrenia (Owned by WCG Verasci, Inc. and contained in the MCCB). He is chief scientific officer of i-Function, Inc. and Scientific Consultant to EMA Wellness, Inc. Dr. Pandina is a full-time employee of Janssen Research & Development, LLC, and a Johnson & Johnson stockholder. Dr. Busner is a full-time employee of Signant Health and may own stock/stock equity. Dr. Jimenez-Shahed is a consultant for Teva, Nuvelution, and Signant Health. Drs. Taylor and Grabb have no conflicts of interest related to the content of this article.

Innov Clin Neurosci. 2023;20(1–3):13–17.


Abstract

This paper expands upon a session, entitled, Special Challenges in Pediatric Drug Development, that was presented as part of a two-day meeting on Pediatric Drug Development at the International Society for Central Nervous System (CNS) Clinical Trials and Methodology (ISCTM) Autumn Conference in Boston, Massachusetts, in October 2020. Drug development in this age group is particularly important because many illnesses have their onset in this age group, many other illnesses that are more common in adults also occur in this time period, and many rare conditions that require special consideration (i.e., orphan conditions) are commonly detected in childhood as well. The special challenges addressed by our speakers in this session were cognitive and functional capacity assessment, challenges of recruitment and assessment of children for research and development of appropriate biomarkers for use in child populations, and the special challenges in training raters to address symptoms in pediatric populations. The speakers have written summaries of their talks. The session’s lead chair was Philip D. Harvey, PhD, who wrote introductory and closing comments. This paper should serve as an expert-informed reference to those interested in and involved in addressing the special challenges facing those involved in CNS pediatric drug development.

Keywords: Pediatric drug development, CNS pediatric psychiatric drug development, pediatric neurologic drug development, pediatric clinical trials


Introductory Comments—Philip D. Harvey, PhD 

This half-day session at the Boston, Massachusetts, International Society for Central Nervous System (CNS) Clinical Trials and Methodology (ISCTM) meeting in October 2020 addressed the challenges of pediatric drug development. Drug development in this age group is particularly important because many illnesses have their onset in pediatric patients, many other illnesses that are more common in adults also occur at this time, and many rare conditions that require special consideration (i.e., orphan conditions) are commonly detected in childhood.

There are multiple considerations for pediatric drug development that were covered in the overall seminar, and in this first session, three were addressed: cognitive and functional capacity assessment, challenges of recruitment and assessment of children for research and development of appropriate biomarkers for use in child populations, and the special challenges in training raters to address symptoms in pediatric populations.

Although the United States (US) National Institutes of Health (NIH) defines childhood as occurring between 0 and 17 years of age, there are multiple developmental issues associated with cognitive functioning assessments during this time. As described by our first presenter, Prof. Gerry Taylor, assessment of children does not rely on downward extension of adult tests. For example, specific developmental challenges make normative standards very challenging in earlier childhood. Normative cognitive performance based on standards for very early childhood would reflect major deficiencies in performance in older children. Another major issue noted by Prof. Taylor is that children are more challenging to assess than adults, and the interface between behavioral problems interfering with valid assessments is much closer in children than in adults. Finally, assessment of functional skills, as typically required by the US Food and Drug Administration (FDA), is even more challenging because many behaviors that would be important to assess are the product of experiences and can be influenced by environmental factors. As childhood behavioral problems are also more commonly detected in underprivileged populations, the challenge of experience is important.

An additional important factor is the development of appropriate biomarkers and biomarker normative standards for children in research. As in cognitive assessments, biomarkers might be different in children and might also be affected by developmental issues. One of the precompetitive consortia described by Dr. Grabb was awarded to Yale University by the National Institute of Mental Health (NIMH) and has established collaborations across the world.

A primary goal of this consortium is to develop understandings and normative standards for pharmacokinetic (PK) and pharmacodynamic (PD) measurements. Such measurements are critical for the development of dosing standards and pursuit of safety across clinical trials. Additionally, the NIH was supporting another consortium, the North American Prodrome Longitudinal Study (NAPLS) for clinical high risk (CHR) for psychosis. The NAPLS project is aimed at prevention of the development of psychosis and has developed a risk calculator. This calculator draws on both behavioral and biomarker features that can then target the highest of the high-risk populations for timely and effective interventions.

The final presentation addressed another important topic in pediatric drug development, rater training. As noted by Prof. Taylor, children are harder to test than adults, and they are also harder to rate. As also noted by Dr. Jimenez-Shahed, because there are fewer pediatric drug trials, it is also likely that raters will be less experienced than those for adult drug trials. Thus, training raters is critically important, and several different standards and strategies are provided for the successful conduct of pediatric drug development trials. Also important is the idea that, in many cases, information will be obtained from parents or guardians, which might lead to multidirectional response biases (e.g., exaggeration in order to enter a trial, underreporting because of concerns about stigmatizing the child, etc.). A further concern is that when children are used as the informant, they might also manifest excessive placebo responses in an effort to please the examiner.

Thus, across these three presentations, both problems and potential solutions are presented. As a result, our understanding of how to approach pediatric drug development is improved, and future efforts should be informed by this information.

Measurement of Cognition and Behavior in Pediatric Drug Trials—H. Gerry Taylor, PhD

Cognitive assessments of children may serve as primary outcomes of drug trials or as secondary outcomes of trials focusing on changes in behavior and other aspects of daily functioning.1 These assessments may be fit for purpose in drug trials that target cognitive change and may provide efficacy and safety information essential for supporting use of the drug. Cognitive measures may also be especially sensitive to drug effects, provide clues regarding the source of behavioral change or, by virtue of evidence for the neural systems subserving particular cognitive skills, shed light on mechanisms of drug effects.

Substantial progress has been made in identifying cognitive tests with effective measurement properties for use in drug trials with adults. Progress is exemplified by the success of the Measurement and Treatment Research to Improve Cognition in Schizophrenia (MATRICS) project in achieving consensus on a test battery for use in adult drug trials.2 Factors considered in test selection included acceptable levels of test-retest reliability, minimal practice effects, relation of performance to functional capacity, and practicality and tolerability in administration. 

Several factors have hampered the development of a comparable set of tests for the wider pediatric age range. The primary challenge in conducting trials with children is that abilities change dramatically with age, with tests appropriate for infants and young children unsuitable for older children. Performance-based measures of infant development engage the child with play materials and in language and motor tasks that would not be useful in differentiating skill levels at later ages. Similarly, performance-based assessments of cognitive abilities for preschool children direct more focus to earlier-emerging skills than do versions of these test batteries designed for school-age youth. Including children of disparate ages in a single drug trial typically involves the use of multiple age-specific measures, precluding analysis of outcomes across age levels.3 A further challenge is that children can be more difficult to test than adults. Examiners need to have experience in building rapport with children, providing appropriate levels of redirection, and coping with limited attention spans. 

Performance-based testing and ratings by parents or teachers are the two primary approaches to assessment in pediatric drug trials. Clinician judgments of child cognition or behavior based on observations or interviews are of potential value but are more rarely implemented, and child self-reports are of limited value in assessing younger children and those with disabilities. Performance-based measures have several benefits over observer reports in assessing drug effects. Tests specify stimulus and demand conditions, as well as scoring procedures, making them more objective than observer ratings. Ratings of behaviors exhibited in everyday settings are more subjective and are likely to be affected by the nature of the setting, though they have the potential to provide valuable information on the effects of drugs on daily functioning. Few performance-based tests of functional capacity (i.e., those similar to tasks encountered in daily living) are available, but development of these measures would allow for assessment of the ecological validity of more traditional cognitive testing.

The psychometric or measurement properties important to selecting cognitive tests for children are similar to those for adults. Additional considerations include the level of engagement and interest in the tasks and the applicability of measures to multiple ages and a wide range of abilities. Tests of fluid cognitive ability are likely to be the most sensitive to drug-related changes, such as those assessing executive function (attention and planning), memory and learning, speed of information processing, and visual motor ability. Numerous computer and paper-and-pencil tests are currently available to assess these abilities, but further research is required to evaluate their applicability to the age range and clinical population under investigation. Modifications of existing procedures or the use of more experimental ones also deserves consideration. Efforts to achieve consensus among experts in designing assessment batteries for pediatric drug trials would also be useful in establishing more uniform standards for these assessments and creating common data elements. 

Securing Patients: Value of Precompetitive Consortia—Meg Grabb, PhD

The NIMH recognizes the need to treat patients early in the course of a psychiatric disorder. Oftentimes, the appropriate target period for intervention is in childhood or early adolescence. The most common path to regulatory approval is to test compounds previously tested in adult populations, then assess the younger age group as appropriate. Typically, bridging studies are needed to establish the PK and PD effects of drugs and determine safety and tolerability. However, the tools and methods needed to perform early-stage PK/PD pediatric bridging studies for psychiatric indications are still lacking. Additionally, objective measures are needed to identify more homogeneous subgroups as surrogates of clinical symptoms and function, as well as to identify subgroup(s) for trial inclusion that best associate their clinical deficit with a compound’s mechanism of action. Therefore, there is an urgent need to develop and validate CNS-based biomarkers in psychiatry with a focus on pediatric populations. 

Numerous research studies have described exploratory biomarkers in psychiatric populations over the last 20 years, primarily in adult populations. Still, biomarker discovery in pediatric populations is growing. The key bottleneck is moving from exploratory observational studies in smaller cohorts to large, multisite studies designed for replication, research assay optimization, and technical/clinical validation. This is due to a number of challenges to the analytic approach in biomarker development, ranging from limits in understanding the disease pathogenesis to complex characteristics of the disease (e.g., limited brain access, age-related changes in behavior of a biomarker, or phenotypic heterogeneity of the disorder). These challenges require a collaborative framework to overcome them, so that academics, industry, government, and foundations can work together in a precompetitive space to identify and agree upon the approach needed to perform rigorous biomarker validation for use in Phase Ib/IIa clinical trials in pediatric populations. Specifically, precompetitive consortia establish agreement on biological targets to select, identify measures to include (with an understanding of their technical limits), establish approaches for standardization and harmonization, and define a strategy for technical and clinical validation. Methods, protocols, and data are then made publicly available so that compounds can be tested in PK/PD trials using established standards. This proceeding highlights two examples of precompetitive consortia, supported by the NIMH, focused on pediatric biomarker development, the Autism Biomarkers Consortium for Clinical Trials (ABC-CT) and NAPLS for CHR for psychosis. 

The ABC-CT project was awarded by the NIH to Yale University (Primary investigator: James McPartland, PhD), and then approved as a Foundation for NIH Biomarkers Consortium (FNIH BC) project in 2015. The FNIH BC model was chosen for this project because it enables public-private partnerships, has established experience in large biomarker efforts including CNS disorders, and supports FDA involvement. Additionally, in 2012, the FNIH BC Neuroscience Steering Committee’s Autism Spectrum Disorder (ASD) Biomarker Working Group had laid the groundwork for pursuing biomarker validation by performing a review of published exploratory biomarkers associated with ASD and assessed replicability, feasibility of data collection in pediatric subjects, cohort sizes, and analytic approach. They determined that electroencephalogram (EEG) and eye tracking were prime candidates for further investment. Under the FNIH BC umbrella, the ABC-CT consortium was formed, comprised of the Eunice Kennedy Shriver National Institute of Child Health and Human Development (NICHD), the NIMH, the National Institute of Neurological Disorders and Stroke (NINDS), Simons Foundation, Janssen, the European Autism Interventions—A Multicentre Study for Developing New Medications (EU AIMS), and the FDA; each group contributed scientifically, providing support, data sharing, and/or regulatory expertise. The objectives of ABC-CT were to evaluate candidate biomarkers, including EEG and eye tracking, for use in future clinical trials in children with ASD aged 6 to 11 years by assessing a variety of technical and clinical factors, including feasibility of data collection, reliability across sites/over time, developmental stability, and sensitivity to change, across five data collection sites. The biomarker’s potential context of use in trials was also assessed and compared to conventional clinician and caregiver assessments.4 The validation approach was vetted by the consortium and deliberately staged such that poor performing biomarkers in the initial feasibility study were removed from further vetting; the goal was to refine as they went. Following the small feasibility study (n=25 subjects with ASD, 25 typical developing [TD] subjects), the main study (n=200 subjects with ASD; 75=TD subjects; baseline, 6-week, and 6-month assessments) included an interim analysis to further focus on the most promising biomarkers.5 Additional details describing the protocol and consortium goals have been described in other publications.6 The overall vetting effort narrowed down the measures to two prognostic biomarkers that remained stable for 24 weeks. The first was a single EEG measure, N170 latency, an event-related potential in response to faces that presented as a delayed response in a subset of subjects with ASD. The second was an eye tracking composite of measures, associated with social scenes, called oculomotor index (OMI), which identifies a subgroup of children with ASD that spend less time looking at social information. 

 An additional consortia model funded by the NIMH is the NAPLS study, which focuses on developing methods to better predict transition of CHR patients to first episode of psychosis so that treatments can be delivered optimally, to the right patients at the right time. The NAPLS consortium is made up of nine research sites and the NIMH; data are collected from CHR subjects (12–35 years old) every few months for up to two years and include neurocognitive, hormonal, blood-based (deoxyribonucleic acid [DNA], ribonucleic acid [RNA], proteomics), electrophysiological, neuroimaging, clinical, demographic, and psychosocial assessments. Several data publications have resulted from the NAPLS consortium, including the establishment of the first individual risk calculator for psychosis (IRC-P) based on the behavioral, cognitive, and demographic measures from 596 subjects.7 New sample cohorts have been studied to refine and validate the prediction models for psychosis.8 Additional studies are focused on defining heterogeneity by determining the preonset trajectories of gray matter decline and disrupted functional brain connectivity in CHR individuals who develop psychosis and identifying inflammatory and plasticity mechanisms associated with the transition to psychosis. Ultimately, these findings could be used to stratify subjects in future drug trials not just based on a subject’s likelihood of advancing to full psychosis, but also by incorporating an individual’s specific pathology, which could then be mechanistically targeted pharmacologically. Importantly, other networks have been established in the US and Europe with similar clinical goals; these networks have joined forces under the Harmonization of At-Risk Multisite Observational Networks for Youth (HARMONY) consortium to synchronize their clinical and biomarker protocols and quality control/quality assurance (QC/QA) procedures, further refine and test prediction algorithms, and analyze a common set of measures across the four networks. 

To further advance the CHR endeavor on a much larger scale, the FNIH BC and NIH recently launched a new public-private partnership as part of the Accelerating Medicines Partnership (AMP) for Schizophrenia program to focus on developing tools (e.g., biological markers, clinical endpoints, and measures that predict trajectories and outcomes) to enable testing of treatments in CHR patients, with the goal of early intervention in schizophrenia. AMP Schizophrenia encompasses the NIH; private and nonprofit partners; two international research networks led by Scott Woods, MD (Yale University), Carrie Bearden, PhD (University of California, Los Angeles), and John Kane, MD (Hofstra/Northwell), and Barnaby Nelson, PhD, and Patrick McGorry, MD, PhD (University of Melbourne); and a data processing, analysis, and coordination center led by Martha Shenton, PhD (Harvard University), and Rene Kahn, MD, PhD (Mount Sinai). With this impressive collaborative model, treatment development should be accelerated and costs lowered through the establishment of a rigorous biomarker validation process, identification of CHR subgroups, and the implementation of early-stage testing of therapies that biologically target subgroups of CHR patients. Biomarker and clinical data will be collected in a large international cohort of CHR young people spanning 42 sites across North America, Australia, Europe, and Asia (12–30 years of age, n=1,977), and participants will be followed for two years. Age-matched healthy control (HC) participants (n=555) will complete screening and baseline assessments and a subset (5 per site) will complete the Month 2 visit. All data management, processing, and analytics are centralized. This is the first AMP initiative focused on a neuropsychiatric disorder and featuring pediatric and young adult cohorts to objectively identify patients most likely to progress to psychosis. The results can enable unprecedented, large-scale data analyses and early pharmacologic intervention in schizophrenia. 

The outcome of these precompetitive consortia to date are the identification of promising biomarkers for use as prognostics (prognostic biomarker definition: a biomarker used to identify likelihood of a clinical event, disease recurrence, or progression in patients who have the disease or medical condition of interest4) in future clinical trials. Letters of Intent for the ABC-CT N170 latency and OMI measures, as well as the NAPLS IRC-P, have been accepted by the FDA Biomarker Qualification Program, which sets the stage for receiving guidance from the FDA on the refinement and further validation of the measures, including the analytic plans.9 If biomarker qualification is obtained, based on successful validation outcomes, the biomarkers may be approved for their intended context of use in future drug trials and available for all drug developers to apply. These efforts have begun to set the stage for future mechanism-based, pediatric, pharmacologic trials; enrich trial cohorts that best associate with a compound’s mechanism of action; define clinical deficits more objectively; establish age-specific PD measures; and better predict subject decline in clinical status over time. Investments in pediatric biomarker research through precompetitive consortia models will play a critical role in biomarker validation and, ultimately, in developing pharmacologic treatments designed for optimized, targeted delivery to individuals earlier in the disease process.

Rater Training for Pediatric Trials—Joohi Jimenez-Shahed, MD

 Rater training is a well-recognized approach to minimizing inaccuracy and variability of clinical outcome assessments. Outcome measures can contain elements of subjectivity and can be influenced by rater judgment, training history, and motivations.10 Particularly in the case of trials involving CNS disorders, many neurology and psychiatry outcome measures rely heavily on clinician judgment. Further challenges to CNS pediatric trials also indicate the need for a well-constructed rater training program. For example, inter- and intrarater reliability can be affected by lack of standardization in certain disease states or rater drift. Accuracy of ratings can also be a problem, especially when there are less experienced investigators or raters, leading to lower concordance with an expert rating or gold standard.

Particularly in the case of CNS disorders, raters might have limited clinical experience and/or training with the selected rating instruments. In turn, the instruments themselves require a rater with sufficient clinical experience to recognize the range of pathology or require knowledge about administration and scoring conventions to achieve ratings consistency. Further challenges in the pediatric population exist. For example, children with CNS disorders might lack the language or cognitive abilities to fully describe their symptoms. As a consequence, raters must integrate input from additional sources, such as parents, caregivers, teachers, or other stakeholders.11 Children who are able to report symptoms may be particularly susceptible to placebo response, as they can be eager to show good cooperation. 

While some of these challenges are inherent to the disease state, rater training and certification programs should aim to appropriately identify well qualified raters who meet minimum qualifications. Approaches to rater training can include didactic, practical, or applied methodologies. In a review of all published descriptions of rater training on any clinical outcome measure, which also included data on reliability and accuracy, Sadler et al10 found that among 29 studies, the majority used a combination of training techniques, which resulted in greater numbers of trials reporting improvements in rating concordance. Additional work has shown that rater training can improve rater competency, regardless of previous clinical experiences, especially with a greater number of training sessions.12 Consensus recommendations on rater training and certification programs have been published and are equally applicable to trials involving pediatric populations.13

Additional recommendations for rater training include the dissemination of a standard set of scoring conventions, use of a structured or semi-structured interview guide in order to standardize rating scale administration, didactic training that assesses the trainee’s conceptual understanding of scoring conventions, and inclusion of applied training wherein observation and feedback are provided to the trainee or testing of applied skills is conducted.14 Pre- and posttesting of rater concordance and consistency can serve to test the efficacy of the training intervention, while posttraining monitoring (including recalibration exercises and “refresher” training) can help maintain interview quality and rater drift. 

Discussion—Philip D. Harvey, PhD

The discussion following the session underscored the points made by the presenters. Dr. Tiffany Farchione, an FDA representative, made it clear that efforts to match tests to age-related developmental factors is consistent with the FDA’s requirement that assessments be fit for purpose, in this case suitable for the age and developmental level of the participants. The needs for adaptation of assessments as children transverse developmental milestones was noted by Dr. Luca Pani, formerly of the European Medicines Agency (EMA). For example, if the long-term safety study is three years, assessments may have to deploy different normative standards for the follow-up assessments. Audience members asked about what the strata were for application of norms; this will need to be specified prior to the trial being launched. It was also noted that there may be behavioral indicators that are age specific. For instance, in some studies, gross abnormalities, such as lack of muscle tone, and related issues, such as maintaining physical position, and related functional acts, such as head control or swallowing, may be outcome measures.

Biomarker assessments were discussed largely in terms of feasibility. Dr. Grabb was asked whether magnetic resonance imaging (MRI) or EEG data would still be an important outcome if the participant had to be sedated in order to collect the data. Her response was that biomarker selection should consider the demands of the assessment at the different developmental levels.

Rater training also attracted a few comments. Dr. Farchione noted that the FDA would commonly ask for documentation of training, particularly in cases where assessments were less commonly used. Concerns were raised regarding several issues. One such question was, do qualifications need to be repeated for each trial or is there the possibility of cross-sponsor certifications? There was a discussion about the fact that even trained raters may need to be recalibrated, even if not retrained, between trials. If trials are looking at improvements at different levels of impairment, based on symptoms, cognition, or physical functioning, these recalibration efforts may be more important.

A general discussion point was the issue of privacy protections. Certain regulations would currently prevent the pooling of even deidentified data, even with parental consent. How much does that kind of regulation protect the participants, and how much does it prevent them from participating in important trials? Furthermore, pooling data has the potential to provide guidance on the safety of interventions that cannot be obtained from a single study. Rare, but serious or even lethal, side effects may not be apparent in initial trials, particularly individual Phase II trials. The focus on protection may have elements that increase risk to individuals with certain diseases, who may later be poised to participate in clinical trials without adequate, wide-ranging safety data to be examined.

Authors’ Note

Drs. Harvey, Pandina, and Busner served as Co-chairs of the Special Challenges in Pediatrics Session. Drs. Taylor, Grabb, and Jimenez-Shahed served as invited speakers.

References

  1. Richardson E, Burnell J, Adams HR, et al. Developing and implementing performance outcome assessments: evidentiary, methodologic, and operational consideration. Ther Innov Regul Sci. 2019;53(1):146–153.
  2. Nuechterlein KH, Green MF Kern RS, et al. The MATRICS Consensus Cognitive Battery: part 1: test selection, reliability, and validity. Am J Psychiatry. 2008;165(2):203–213.
  3. Edgar CJ, Wesnes KA. Cognition assessment in paediatric clinical trials. Drug Discov Today. 2008;13(1–2):79–85.
  4. FDA-NIH Biomarker Working Group. BEST (Biomarkers, EndpointS, and other Tools) Resource. Food and Drug Administration (US); 2016. Co-published by National Institutes of Health (US).
  5. Webb SJ, Shic F, Murias M, et al. Biomarker acquisition and quality control for multi-site studies: the Autism Biomarkers Consortium for Clinical Trials. Front Integr Neurosci. 2020;13:71. 
  6. McPartland JC, Bernier RA, Jeste SS, et al. The Autism Biomarkers Consortium for Clinical Trials (ABC-CT): scientific context, study design, and progress toward biomarker qualification. Front Integr Neurosci. 2020;14:16. 
  7. Cannon TD, Yu C, Addington J, et al. An individualized risk calculator for research in prodromal psychosis. Am J Psychiatry. 2016;173(10):980–988. 
  8. Cannon TD, Cadenhead K, Cornblatt B, et al. Prediction of psychosis in youth at high clinical risk: a multisite longitudinal study in North America. Arch Gen Psychiatry. 2008;65(1):
    28–37. 
  9. United States Food and Drug Administration. About biomarkers and qualification. Current as of 7 Jul 2021. https://www.fda.gov/drugs/biomarker-qualification-program/about-biomarkers-and-qualification. Accessed 23 Jan 2023.
  10. Sadler ME, Yamamoto RT, Khurana L, Dallabrida SM. The impact of rater training on clinical outcomes assessment data: a literature review. Int J Clin Trials. 2017;4(3):1–10.
  11. Busner J. Challenges in child and adolescent psychopharmacology clinical trials. Child Adolesc Psychopharmacol News. 2013;18(3):1–4.
  12. Targum SD. Evaluating rater competency for CNS clinical trials. J Clin Psychopharmacol. 2006;26(3):308–310.
  13. Daniel DG, Opler M, Wise-Rankovic A, Kalali A. Consensus recommendations on rater training and certification. Innov Clin Neurosci. 2014;11(11–12):10–13. 
  14. Kobak KA, Engelhardt N, Williams JB, Lipsitz JD. Rater training in multicenter clinical trials: issues and recommendations. J Clin Psychopharmacol. 2004;24(2):113–117.