Performing risk assessments for the risk-based monitoring of clinical trials

Risk assessment is an integral part of risk management. In clinical trials, it serves to lay the foundation of risk-based monitoring processes. Regulators currently encourage its use to improve the safety and the quality of clinical trials. In practice, assessing risks is difficult but clinical research professionals must become acquainted with the concept of risk assessment in order to reap the full potential of risk-based monitoring.

Regulatory Perspective

The FDA’s most recent guidance is fairly specific about the importance of conducting a risk assessment to identify risk and develop a monitoring plan:

The risk assessment serves to identify and understand the nature, sources, likelihood of detection, and potential causes of risks that could affect the collection of critical data or performance of critical processes. The risk assessment informs the development of a monitoring plan and may also support efforts to manage risks across a clinical investigation or across a product’s development program.

Accordingly, a risk assessment serves to identify risk factors which can undermine the safety, quality and integrity of a given study. It also serves to define the nature of risk factors and develop strategies to either eliminate them or mitigate them. The output of the risk assessment specifically provides elements for the development of a risk management plan which constitutes the blueprint of the risk-based monitoring process. This plan namely includes (1) rationales for choosing to monitor different risk factors, (2) details about the calculation of risk factors’ metrics, (3) limits or tolerance thresholds associated with each metric and (4) actions to be considered when metrics fall outside their set limits. The following steps can be performed, more or less in the following order, to produce a risk management plan:

1. Identify processes that are critical to subject safety, trial integrity and data quality
2. Assess the vulnerability of the processes to specific risk factors
3. Determine the likelihood and impact of each risk factor
4. Find ways to eliminate risk factors or reduce their impacts
   • Identify information available to monitor risk
   • Determine how Key Risk Indicators (KRI) metrics are calculated from the information available
   • Determine limits beyond which site-specific KRI metrics should be investigated
   • Determine what remedial actions should be considered when KRI metrics fall beyond limits
5. Continually re-assess risk factors and risk limits

Here are some more details about each step:

Step 1- Identify processes that are critical to subject safety, trial integrity and data quality

The risk assessment should be done by a competent team of individuals who have a good working knowledge of a given protocol and the critical processes that support subject safety, trial integrity and data quality. Indeed, being aware of things such as the enrolment process, the procedures to be performed, the general health status of subjects and how data are to be collected is essential for identifying critical processes.

Step 2 - Assess the vulnerability of the processes to specific risk factors

The definition of risk is the possibility that an event will occur and adversely affect the achievement of objectives. In clinical research, any event that can compromises critical processes with an impact on subject safety, trial integrity and data quality qualifies as a risk factor. For example, subject enrolment is a critical process that affects the integrity of all trials and the risk factors that can adversely affect it include an insufficient patient pool, a deficient informed consent process, an inadequate publicity campaign, eligibility criteria that are too restrictive, etc.

Step 3 - Determine the likelihood and impact of each risk factor

After identifying a study’s risk factors, their potential severity of impacts and likelihood of occurrence should be estimated. In practice, stakeholders hold diverse perspectives regarding impacts and likelihoods of risk factors. Different people have different risk tolerance and they do not accept risks in isolation but in the context of the associated benefits. Indeed, acceptable risk is shorthand for a voluntarily assumed risk accompanied by anticipated benefits. The process of risk assessment thus involves reconciling stakeholders’ values. With a risk matrix such as the one below, subjective estimates of risks can be translated into quantitative measures. There are many different types of risk matrix but the figure below represents the most basic one. Transcelerates’s Risk Assessment and Categorization Tool (RACT) namely uses this type of matrix to compute of risk scores.

Risk Importance

The risk scores generated using a risk matrix can be used to rank the different risk factors according to their respective “importance”. A risk factor’s importance is commensurate with the need to control it. It also serves as a gauge for the intensity of the contingency actions to be deployed in the event that a risk metric falls outside of its set limits. 

Risk factors’ importance change as a trial progresses. For example, risks to the enrolment process are very important at the beginning of a trial but they become less important after the recruitment is over. On the other hand, the risks to data entry and query resolution increase in importance as the trial progresses because they have a direct impact on the time to database lock. As such, the trial’s timeline must be considered when determining the importance of risk factors.

Step 4 - Find ways to eliminate risk factors or reduce their impacts

Risk factors may be avoided or their impact made insignificant by adjusting processes. For example, if the primary endpoints are to be collected in an EDC system, designing effective data entry edit checks can almost eliminate the risk that entry errors pose to data quality. Providing an excel tool to study coordinators for the scheduling of visits can reduce the risk of subjects visits being scheduled out of the allowed time window. Making sure that sites serve an adequate number of potential subjects and that the publicity campaign targets the right audience can reduce the risk of insufficient enrolment.

Modifying the protocol can also serve to avoid or reduce risks. For example, eligibility criteria may be changed if they compromise enrolment. The schedule of assessment may be changed if it is likely to cause missed visits. In reality though, once a protocol is written and sponsors have approved it, changing it is not always an option for the people who are to implement it. Therefore, unless it is absolutely necessary to change the protocol to avoid obvious risks, clinical operations along with data management must devise strategies to mitigate their potential impact.

4.2. Identify information available to monitor risk

Some risk factors cannot be eliminated and Key Risk Indicators (KRI) related to those must be calculated and monitored throughout the trial. As such, information required for the calculation of those KRIs must be collected on a periodic basis. Sources of information pertaining to risk include EDC systems database and logs kept by clinical operations in CTMS systems and Excel spreadsheets.

Because risk-based monitoring implies that efforts are concentrated on what is important, KRIs must be selected in a way that ensures resources efficiency. Typically, around ten KRIs should be selected for any given trials. Fortunately, the majority of KRIs can be taken from a list of common KRIs and only a few additional trial-specific KRIs must added to the list for a given trial.

Site Level Risk

Some sites represent higher risk than other due to their different internal processes and experiences in clinical research. As such, site-specific information acquired from previous experience and site initiation visits should be taken into consideration when determining the monitoring approach for individual sites. The impression gathered from periodic on-site monitoring visits also represent useful and meaningful qualitative information.

4.3. Determine how Key Risk Indicators are calculated from the information available

The computation of KRIs must be carefully thought about to ensure that they allow to compare sites without bias. As shown in the table below, calculating screen-failure metrics can be done in different ways but one way avoids generating false risk signals and missing risk signals:

4.4. Determine limits beyond which site-specific KRI metrics should be investigated

Limits, also known as “thresholds” or “tolerance level”, are metric values beyond which KRI metrics may be considered abnormal and warrant investigation. Those must be carefully chosen to be able to detect risk without triggering too many false risk signals. Also limits should be adjusted as the study progresses and more data becomes available for analysis. See the post Setting Limits in Centralized Monitoring for more insights regarding the choice of KRI limits.

4.5. Determine what remedial actions should be considered when KRI metrics fall beyond limits

A number of situations may result in KRI metrics falling beyond their set limits. As such, risk signals must be managed with adaptive responses. Here are some mitigation actions to be considered in response to different risk signals. As discussed in the posts Centralized Monitoring False Risk Signals and Planning for Change in Centralized Monitoring, the analysis of risk signals often reveals that no action is required or a very specific and customized action is required. Still, early planning is the crux of risk management.

Step 5. Continually re-assess risk factors and risk limits

The risk assessment is a cyclical process. Because it is performed before the trial begins, it lacks the contexts in which risks are encountered later in reality. Therefore KRIs, limits and mitigation actions must be continuously reviewed to ensure that they are appropriately selected and the risk-based monitoring process is continuously improved upon.

Risk Assessment Tools

Tools are available to facilitate the execution of risk management in clinical trial. Trancelerate’s Risk Assessment and Categorization Tool (RACT) is an excel workbook that enables the identification of risk factors and their ranking according to their relative importance. XLSMetrics’ RI Calculator is an excel workbook that enables the monitoring of KRIs as trials progress. Excel may appear archaic but it is a robust solution for the calculation of all sorts of metrics. It is also flexible enough to be customized into the risk-based monitoring strategy of any trial.