Raters play a critical role in ensuring the validity and reliability of study results. The overall objective of rating is to minimize variability in the assessment of study outcomes by following standardized protocols. This consistency allows for accurate comparisons between treatment groups and reduces the potential for bias. In central nervous system (CNS) clinical trials, rating is important because CNS disorders can manifest differently among individuals, making it challenging to consistently assess treatment efficacy. Moreover, mood, personality, and cognitive factors may influence subjective reporting by study participants.
Approaches to Rating in Clinical Trials
Raters are trained individuals who assess the symptoms, signs, and outcomes of clinical trial participants using standardized assessment tools. Their role is to provide objective, reliable measurements of treatment response and factors such as disease severity, functional status, or quality of life in the study population. In clinical trials, there are two approaches to rating which differ in location and number of raters:
- Centralized rating, which involves having a single group of raters, often at a central location, assess the symptoms and outcomes of study participants using standardized assessment tools. This approach aims to minimize the potential for rater bias and variability.
- Decentralized rating, which involves having raters at multiple sites or locations. This approach aims to increase the efficiency of data collection by distributing the workload across multiple raters and may be more practical for studies that involve large geographic areas or diverse populations.
Each rating approach has advantages and disadvantages, and the choice of which approach to use depends on the specific needs of the clinical trial (see Table 1). For example, centralized rating may be preferred for studies that require complex assessments or a high level of standardization and consistency. Decentralized rating may be preferable when flexibility and efficiency of data collection are required.
Pros and Cons of Rating Approaches (Table 1)
| Advantages | Disadvantages | |
|---|---|---|
| Decentralized Rating |
|
|
| Centralized Rating |
|
|
Key Considerations for Selecting a Rating Approach
When determining which type of rating is most appropriate for a clinical trial, consider:
Indication under investigation
Complexity of a disease and its symptomatology can influence the rating process. For CNS disorders such as Parkinson’s disease that require specialized training and experience to evaluate symptom severity accurately, centralized rating may be more appropriate. This is also true for studies where the ratings are primary or secondary endpoints.
Study design
The frequency of evaluations and the feasibility of performing those evaluations remotely also affect the choice of rating approach. For studies that require frequent evaluations or in-person assessments, decentralized rating may be the most cost-effective option.
Investigator experience
Experienced investigators may have a better understanding of the disease under investigation and the potential side effects or complications of the treatment being tested, making them better equipped to identify and measure the outcomes of interest. They may also be more familiar with the tools and methods used to assess these outcomes, which helps ensure that the data collected is reliable and accurate with either rating approach.
Site experience
Experienced sites may have more experience working with the patient population of interest and more resources and infrastructure in place to ensure the quality and accuracy of data collected with decentralized rating. For less experienced sites without expert raters, a centralized approach may accelerate study startup as hiring, training, testing, and overseeing local raters may be more costly and time-consuming.
Rating scale(s) used
Rating scales vary in complexity and level of validation or availability of expert guidelines or published literature to support interpretation. One study evaluating different versions of the Hamilton Rating Scale for Depression (HAM-D) found that these versions varied in their clinimetric properties and concluded that the most appropriate version for a clinical trial would depend on rater experience, study design and objectives, and clinical characteristics of the population under evaluation.i
For rating scales that require visual assessment, centralized rating has the advantage of reducing variability in ratings since all raters are trained and supervised in a standardized fashion. For rating scales that are complex and require significant and ongoing training and oversight to ensure compliance and consistency, centralized rating may be more appropriate.
Likelihood of placebo response, response bias, and rater bias
The characteristics of both patients and raters can impact rating measurements. With patients, placebo response occurs when the positive expectation of improvement impacts self-reported outcomes. In clinical trials of antidepressant medications, placebo response is substantial.ii Response bias is introduced when patients answer rating questionnaire items with the response, they perceive to be most desirable to the rater or study staff. With raters, bias can be introduced when the rater’s underlying beliefs regarding the treatment under investigation influence their rating. When either placebo response, response bias, or rater bias is expected to be high, utilizing centralized raters to perform both the screening and outcome measures of a study may minimize bias and maximize consistency.
For some studies, a hybrid approach that combines both centralized and decentralized rating may be appropriate. For example, a trial could use local raters to screen for eligibility and central raters to assess for efficacy, provided that all raters receive the same training and testing. In CNS conditions such as Alzheimer’s disease where telemedicine examinations correlate well with in-person evaluations, using the patient’s own physician—under the direction of two central neurologists—to perform rating assessments maintains consistency and may help to accelerate enrollment.iii
Tips for Implementing Centralized Rating
Implementing a centralized rating approach in a clinical trial requires careful planning, training, and monitoring, as well as multi-stakeholder collaboration to define the workflow and supporting processes. Once the outcome measure(s) to be assessed have been defined in the study protocol, implementation involves:
- Identifying a group of experienced, qualified raters with expertise in the outcome measure(s) being assessed and experience in conducting centralized ratings.
- Determining technology needs which are necessary to support centralized rating, whether it is electronic patient reported outcomes (ePROs) or electronic clinical outcomes assessments (eCOAs) and selecting qualified vendors.
- Developing and implementing a standardized rater training program that covers the rating scale(s) to be used, including practical training to ensure that all raters apply the rating scale in a consistent manner.
- Conducting a reliability study to assess inter-rater reliability. This involves having multiple raters independently rate a set of standardized cases to evaluate consistency among their individual ratings.
- Monitoring rating quality throughout the study with regular review of a sample of the ratings to ensure there is no rating drift.
- Providing ongoing training, support and supervision for the duration of the study.
Key Takeaway
Rating plays a critical role in CNS trials, providing a standardized and objective method for evaluating the efficacy and safety of investigational treatments. There is no one-size-fits-all approach to rating in CNS trials and determining whether to implement a centralized, decentralized or hybrid approach requires careful consideration of the study objectives and outcome measures, with a focus on optimizing the reliability, validity, and accuracy of results.
Precision for Medicine is experienced in managing rated studies, including both adult and pediatric CNS clinical trials. Through our support of over 60 CNS clinical trial programs and more than 600 CNS commercialization projects, we have developed expertise in the common scales used in CNS and psychiatry and built a network of qualified vendors specializing in rater training and monitoring.
