Raters play a critical role in ensuring the validity and reliability of study results. The overall objective of rating is to minimize variability in the assessment of study outcomes by following standardized protocols. This consistency allows for accurate comparisons between treatment groups and reduces the potential for bias.
In central nervous system (CNS) clinical trials, rating is important because CNS disorders can manifest differently among individuals, making it challenging to consistently assess treatment efficacy. Moreover, mood, personality, and cognitive factors may influence subjective reporting by study participants.
Raters are trained individuals who assess the symptoms, signs, and outcomes of clinical trial participants using standardized assessment tools. Their role is to provide objective, reliable measurements of treatment response and factors such as disease severity, functional status, or quality of life in the study population.
In clinical trials, there are two approaches to rating which differ in location and number of raters:
Each rating approach has advantages and disadvantages, and the choice of which approach to use depends on the specific needs of the clinical trial. For example:
| Advantages | Disadvantages | |
|---|---|---|
| Decentralized Rating | Assessment by a provider that knows the patient, which may be more comfortable for the patient Provision of on-site care, which clinical trial participants may expect or prefer | Difficulty in achieving standardization at a study-wide level Cost and time involved in hiring, training, and maintaining expert raters at each study site |
| Centralized Rating | Increased patient access and convenience, which may help with recruitment and retention Bigger pool of eligible sites, since sites no longer need to have their own expert clinical raters Centralized reporting, which allows rapid identification and remediation of non-compliant visits Well-trained, independent raters, may be more objective in determining eligibility and treatment effectiveness Rating standardization, due to centralized training and calibration which lead to higher levels of intra- and inter-rater reliability Ability to study patient in real-world settings, rather than the clinical trial environment | Fewer face-to-face interactions with study staff Requires good connectivity and technology capability, for both sites and study participants Cost, depending on the study protocol and design |
Effectively Managing Rater Training and Consistency in CNS Trials
When determining which type of rating is most appropriate for a clinical trial, consider:
Complexity of a disease and its symptomatology can influence the rating process. For CNS disorders such as Parkinson’s disease that require specialized training and experience to evaluate symptom severity accurately, centralized rating may be more appropriate. This is also true for studies where the ratings are primary or secondary endpoints.
The frequency of evaluations and the feasibility of performing those evaluations remotely also affect the choice of rating approach. For studies that require frequent evaluations or in-person assessments, decentralized rating may be the most cost-effective option.
Experienced investigators may have a better understanding of the disease under investigation and the potential side effects or complications of the treatment being tested, making them better equipped to identify and measure the outcomes of interest. They may also be more familiar with the tools and methods used to assess these outcomes, which helps ensure that the data collected is reliable and accurate with either rating approach.
Experienced sites may have more experience working with the patient population of interest and more resources and infrastructure in place to ensure the quality and accuracy of data collected with decentralized rating. For less experienced sites without expert raters, a centralized approach may accelerate study startup as hiring, training, testing, and overseeing local raters may be more costly and time-consuming.
Rating scales vary in complexity and level of validation or availability of expert guidelines or published literature to support interpretation. One study evaluating different versions of the Hamilton Rating Scale for Depression (HAM-D) found that these versions varied in their clinimetric properties and concluded that the most appropriate version for a clinical trial would depend on rater experience, study design and objectives, and clinical characteristics of the population under evaluation.i
For rating scales that require visual assessment, centralized rating has the advantage of reducing variability in ratings since all raters are trained and supervised in a standardized fashion. For rating scales that are complex and require significant and ongoing training and oversight to ensure compliance and consistency, centralized rating may be more appropriate.
The characteristics of both patients and raters can impact rating measurements. With patients, placebo response occurs when the positive expectation of improvement impacts self-reported outcomes. In clinical trials of antidepressant medications, placebo response is substantial.ii Response bias is introduced when patients answer rating questionnaire items with the response, they perceive to be most desirable to the rater or study staff.
With raters, bias can be introduced when the rater’s underlying beliefs regarding the treatment under investigation influence their rating. When either placebo response, response bias, or rater bias is expected to be high, utilizing centralized raters to perform both the screening and outcome measures of a study may minimize bias and maximize consistency.
For some studies, a hybrid approach that combines both centralized and decentralized rating may be appropriate.
In CNS conditions such as Alzheimer’s disease where telemedicine examinations correlate well with in-person evaluations, using the patient’s own physician—under the direction of two central neurologists—to perform rating assessments maintains consistency and may help to accelerate enrollment.iii
Implementing a centralized rating approach in a clinical trial requires careful planning, training, and monitoring, as well as multi-stakeholder collaboration to define the workflow and supporting processes. Once the outcome measure(s) to be assessed have been defined in the study protocol, implementation involves:
Rating plays a critical role in CNS trials, providing a standardized and objective method for evaluating the efficacy and safety of investigational treatments. There is no one-size-fits-all approach to rating in CNS trials and determining whether to implement a centralized, decentralized or hybrid approach requires careful consideration of the study objectives and outcome measures, with a focus on optimizing the reliability, validity, and accuracy of results.
Precision for Medicine is experienced in managing rated studies, including both adult and pediatric CNS clinical trials. Through our support of over 60 CNS clinical trial programs and more than 600 CNS commercialization projects, we have developed expertise in the common scales used in CNS and psychiatry and built a network of qualified vendors specializing in rater training and monitoring.
Robert Bauer is an Executive Director of Operational Strategy at Precision with over 20 years of clinical trial operations experience from both the sponsor and CRO perspectives. He has led teams in the conduct of trials from Phase I through IV and has experience across a wide variety of trial designs. Bob has experience across many therapeutic areas some of which include rare diseases, oncology, depression, schizophrenia, and substance use disorders.
Vincent Shortino is a Sr. Clinical Trial Manager, bringing more than 20 years of experience to Precision. His experience spans all major therapeutic areas with a heavy emphasis on Cardiovascular and Rare Indications, across Phases I–IV. He brings solid perspectives inclusive of sites, pharma and CROs from his previous roles of Study Coordinator, Site Director, CRA, and Clinical Trial Manager.
Precision for Medicine is part of the Precision Medicine Group, an integrated team of experts that extends Precision for Medicine’s therapeutic development capabilities beyond approval and into launch strategies, marketing communication, and payer insights. As one company, the Precision Medicine Group helps pharmaceutical and life-sciences clients conquer product development and commercialization challenges in a rapidly evolving environment.
