Raters play a critical role in ensuring the validity and reliability of study results. The overall objective of rating is to minimize variability in the assessment of study outcomes by following standardized protocols. This consistency allows for accurate comparisons between treatment groups and reduces the potential for bias.
In central nervous system (CNS) clinical trials, rating is important because CNS disorders can manifest differently among individuals, making it challenging to consistently assess treatment efficacy. Moreover, mood, personality, and cognitive factors may influence subjective reporting by study participants.
Approaches to Rating in Clinical Trials
Raters are trained individuals who assess the symptoms, signs, and outcomes of clinical trial participants using standardized assessment tools. Their role is to provide objective, reliable measurements of treatment response and factors such as disease severity, functional status, or quality of life in the study population.
In clinical trials, there are two approaches to rating which differ in location and number of raters:
Centralized rating: Involves a single group of raters—often at a central location—assess the symptoms and outcomes of study participants using standardized assessment tools. This approach aims to minimize the potential for rater bias and variability.
Decentralized rating: Involves raters at multiple sites or locations. This approach aims to increase the efficiency of data collection by distributing the workload across multiple raters and may be more practical for studies that involve large geographic areas or diverse populations.
Each rating approach has advantages and disadvantages, and the choice of which approach to use depends on the specific needs of the clinical trial. For example:
Centralized rating: Typically for studies with complex assessments or high levels of standardization and consistency.
Decentralized rating: Useful for times when there is a need for flexible and efficient data collection.
Pros and Cons of Rating Approaches
Assessment by a provider that knows the patient, which may be more comfortable for the patient
Provision of on-site care, which clinical trial participants may expect or prefer
Difficulty in achieving standardization at a study-wide level
Cost and time involved in hiring, training, and maintaining expert raters at each study site
Increased patient access and convenience, which may help with recruitment and retention
Bigger pool of eligible sites, since sites no longer need to have their own expert clinical raters
Centralized reporting, which allows rapid identification and remediation of non-compliant visits
Well-trained, independent raters, may be more objective in determining eligibility and treatment effectiveness
Rating standardization, due to centralized training and calibration which lead to higher levels of intra- and inter-rater reliability
Ability to study patient in real-world settings, rather than the clinical trial environment
Fewer face-to-face interactions with study staff
Requires good connectivity and technology capability, for both sites and study participants
Key Considerations for Selecting a Rating Approach
When determining which type of rating is most appropriate for a clinical trial, consider:
Indication under investigation
Complexity of a disease and its symptomatology can influence the rating process. For CNS disorders such as Parkinson’s disease that require specialized training and experience to evaluate symptom severity accurately, centralized rating may be more appropriate. This is also true for studies where the ratings are primary or secondary endpoints.
The frequency of evaluations and the feasibility of performing those evaluations remotely also affect the choice of rating approach. For studies that require frequent evaluations or in-person assessments, decentralized rating may be the most cost-effective option.
Experienced investigators may have a better understanding of the disease under investigation and the potential side effects or complications of the treatment being tested, making them better equipped to identify and measure the outcomes of interest. They may also be more familiar with the tools and methods used to assess these outcomes, which helps ensure that the data collected is reliable and accurate with either rating approach.
Experienced sites may have more experience working with the patient population of interest and more resources and infrastructure in place to ensure the quality and accuracy of data collected with decentralized rating. For less experienced sites without expert raters, a centralized approach may accelerate study startup as hiring, training, testing, and overseeing local raters may be more costly and time-consuming.
Rating scale(s) used
Rating scales vary in complexity and level of validation or availability of expert guidelines or published literature to support interpretation. One study evaluating different versions of the Hamilton Rating Scale for Depression (HAM-D) found that these versions varied in their clinimetric properties and concluded that the most appropriate version for a clinical trial would depend on rater experience, study design and objectives, and clinical characteristics of the population under evaluation.i
For rating scales that require visual assessment, centralized rating has the advantage of reducing variability in ratings since all raters are trained and supervised in a standardized fashion. For rating scales that are complex and require significant and ongoing training and oversight to ensure compliance and consistency, centralized rating may be more appropriate.
Likelihood of placebo response, response bias, and rater bias
The characteristics of both patients and raters can impact rating measurements. With patients, placebo response occurs when the positive expectation of improvement impacts self-reported outcomes. In clinical trials of antidepressant medications, placebo response is substantial.ii Response bias is introduced when patients answer rating questionnaire items with the response, they perceive to be most desirable to the rater or study staff.
With raters, bias can be introduced when the rater’s underlying beliefs regarding the treatment under investigation influence their rating. When either placebo response, response bias, or rater bias is expected to be high, utilizing centralized raters to perform both the screening and outcome measures of a study may minimize bias and maximize consistency.
For some studies, a hybrid approach that combines both centralized and decentralized rating may be appropriate.
Example: a trial could use local raters to screen for eligibility and central raters to assess for efficacy (provided that all raters receive the same training and testing)
In CNS conditions such as Alzheimer’s disease where telemedicine examinations correlate well with in-person evaluations, using the patient’s own physician—under the direction of two central neurologists—to perform rating assessments maintains consistency and may help to accelerate enrollment.iii
Implementing a centralized rating approach in a clinical trial requires careful planning, training, and monitoring, as well as multi-stakeholder collaboration to define the workflow and supporting processes. Once the outcome measure(s) to be assessed have been defined in the study protocol, implementation involves:
Identifying a group of experienced, qualified raters with expertise in the outcome measure(s) being assessed and experience in conducting centralized ratings.
Determining technology needs which are necessary to support centralized rating, whether it is electronic patient reported outcomes (ePROs) or electronic clinical outcomes assessments (eCOAs) and selecting qualified vendors.
Developing and implementing a standardized rater training program that covers the rating scale(s) to be used, including practical training to ensure that all raters apply the rating scale in a consistent manner.
Conducting a reliability study to assess inter-rater reliability. This involves having multiple raters independently rate a set of standardized cases to evaluate consistency among their individual ratings.
Monitoring rating quality throughout the study with regular review of a sample of the ratings to ensure there is no rating drift.
Providing ongoing training, support and supervision for the duration of the study.
Rating plays a critical role in CNS trials, providing a standardized and objective method for evaluating the efficacy and safety of investigational treatments. There is no one-size-fits-all approach to rating in CNS trials and determining whether to implement a centralized, decentralized or hybrid approach requires careful consideration of the study objectives and outcome measures, with a focus on optimizing the reliability, validity, and accuracy of results.
Precision for Medicine is experienced in managing rated studies, including both adult and pediatric CNS clinical trials. Through our support of over 60 CNS clinical trial programs and more than 600 CNS commercialization projects, we have developed expertise in the common scales used in CNS and psychiatry and built a network of qualified vendors specializing in rater training and monitoring.
Robert Bauer is an Executive Director of Operational Strategy at Precision with over 20 years of clinical trial operations experience from both the sponsor and CRO perspectives. He has led teams in the conduct of trials from Phase I through IV and has experience across a wide variety of trial designs. Bob has experience across many therapeutic areas some of which include rare diseases, oncology, depression, schizophrenia, and substance use disorders.
Vincent Shortino is a Sr. Clinical Trial Manager, bringing more than 20 years of experience to Precision. His experience spans all major therapeutic areas with a heavy emphasis on Cardiovascular and Rare Indications, across Phases I–IV. He brings solid perspectives inclusive of sites, pharma and CROs from his previous roles of Study Coordinator, Site Director, CRA, and Clinical Trial Manager.
Precision for Medicine is part of the Precision Medicine Group, an integrated team of experts that extends Precision for Medicine’s therapeutic development capabilities beyond approval and into launch strategies, marketing communication, and payer insights. As one company, the Precision Medicine Group helps pharmaceutical and life-sciences clients conquer product development and commercialization challenges in a rapidly evolving environment.