When using a centralized monitoring approach, Key Risk Indicators (KRIs) metrics are calculated for individual study sites and compared against limits, also known as “thresholds” or “tolerance level”, beyond which sites may be considered abnormal and warrant investigation. Limit values must be carefully chosen to be able to detect risk without triggering too many false signals. Information relevant to the choice of limits can be obtained by executing a trial’s risk assessment and by calculating site-specific KRIs statistical distributions. Below are some points to consider when choosing limits.
Limits Based on Risk Assessment
Risk identified through the risk assessment should reflect the protocol requirements and the expectations from the clinical team regarding the performance and the quality of investigational sites. For example, if data is expected to be entered in an EDC system within 5 days of being collected, it is justified to set a limit of 5 working days to detect sites that are lagging in data entry. Most KRIs have upper limits but some KRIs may also have lower limits. For example, an upper limit may be set for the AE Rate to detect sites which report an unusually high number of AEs and a lower limit may be set to detect sites at which coordinators are missing or under-reporting AEs.
To get you started on identifying KRIs and choosing limits, see this List of KRIs with their rationales for monitoring and examples of actions to be considered when the KRI metrics falls beyond their set limits.
Normal and Critical Limits
In reality, some sites may fall beyond limits for reasons that do not require immediate action (e.g.: vacations, holidays). Other sites may fall beyond limits for reasons that compromise subject's safety, trial integrity or data quality, in which case prompt action is required. Because most risk tolerance concepts involves risk levels (e.g.: Low, Moderate, High), two types of limit may be set for a given KRI to discriminate sites that generate different levels of risk. Namely, “normal” limits can stand to represent a threshold beyond which a site-specific metric is considered “different” from the rest of the sites. An additional “critical” limit can stand to identify sites’ KRIs that exceed the tolerance level of the clinical team.
Performing a risk assessment is the most efficient way to determine what values should constitute “normal” and “critical” limits and what actions should be considered when a given site KRI metrics trespasses each type of limits. Statistical analysis of KRIs distributions is also very useful once the trial gets going and data become available for centralized monitoring.
Limits Based on Standard Deviations from the Mean
KRIs metrics standard deviations are relevant to setting limits as they represent the real-world variations in site-specific KRI metrics. For example, in a trials where the efficacy of a treatment is still unproven, setting arbitrary limits for the Withdrawal Rate metric is difficult as the number of subjects who will withdraw due to a perceived lack of efficacy is hard to foresee. Therefore, one may use the data collected and choose to set a “normal” limit at a value corresponding to 2 standard deviation from the mean and a “critical” limit at a value corresponding to 3 standard deviation from the mean (statistically speaking, in a normal distribution, approximately 95% of observations fall within 2 standard deviations from the mean and approximately 99% fall within 3 standard deviations from the mean). As such, it is possible to identify sites that are somewhat different as well as sites that are really different from other sites. Using the standard deviation from the mean requires a that a certain amount of data be collected in order to detect risk signals with a significant statistical power. Nevertheless, early analysis can still serve to detect outliers and catch potential problems early. See the article The Basics of Clinical Trial Centralized Monitoring for more information regarding the statistical notions related to centralized monitoring.
Limits Based on Data from Previous Studies
The experience gained from previous studies must be taken into account when setting limits. For instance, calculating the normal AE rate of the population under study using the data from previous studies is a better approach then using the data from the current study. Not only does the low amount of data available at the beginning of a study implies low statistical power, if the treatment investigated causes an increase in the AE rate, flagging sites based only on the current data does not allow to detect a possible shift in the population mean which constitutes a risk signal in itself. Thus, one can monitor subjects’ safety by setting limits based on the data collected in previous studies done on populations that resemble the population under study in terms of age, gender, medical history, concomitant medications, etc.
Reacting to Risk Signals
When a KRI metric value falls beyond it set limit, contacting sites is a good opportunity for sponsor to provide help and support to the sites participating in its investigation. Reprimanding sites for falling beyond limits is rarely constructive and the strained relationship that results only makes it harder to solve current and future problems. Sponsors should try to understand individual situations and try to offer genuine help to sites. For instance, if a site has an elevated average Time to Query Resolution, sending an email telling the site coordinators that they do not meet the sponsor’s expectation is hardly addressing the issue. Looking at the nature of the pending queries and providing instruction on query resolution is much more effective. Indeed, some EDC systems have really useful functions to help resolve queries which coordinators may not be aware of. Investigating individual KRIs metrics that fall beyond limits and providing thoughtful assistance is one of the purpose of the central monitors. As discussed in a previous post Planning for Change in Centralized Monitoring, investigating why KRI metrics fall beyond their set limits and documenting the reasons along with actions taken represent a key aspect of centralized monitoring. Communication skills are very important for keeping stakeholders informed and synergizing the activity of the clinical operations team through the centralized monitoring process.