Recommendations and the Broken KPI

Written by Risk Alive

April 22, 2018

Ronjit Mukherjee
Risk Alive Analytics Inc.
#300, 926 5th Avenue
Calgary, AB, Canada, T2P 0N7
rmukherjee@acm.ca

Allison De Man
Risk Alive Analytics Inc.
#300, 926 5th Avenue
Calgary, AB, Canada, T2P 0N7
ademan@acm.ca

Prepared for Presentation at American Institute of Chemical Engineers

2018 Spring Meeting and 14th Global Congress on Process Safety

Orlando, Florida, April 22 – 25, 2018

Abstract

Process company metrics and CCPS Industry Surveys show that the Percentage of Process Hazard Analysis (PHA) Recommendations Completed can be considered a process safety “Leading KPI” [1]. This type of metric focuses companies on the quantity of PHA recommendations implemented, and not necessarily the risk reduction impact or benefit of the recommendations. As PHAs often identify tens to hundreds of recommendations, the philosophy to implement as many recommendations as possible can lead to unexpected risk exposure, either by not implementing risk critical recommendations, or by not implementing them in a reasonable time frame. This can also lead to inefficient financial expenditures and resource utilization.

Process industries can improve on these current practices and focus on implementing the most risk reducing recommendations, instead of implementing “as many as possible”.

Three key elements which will assist in this change of philosophy from quantity to quality, and in turn ensure workers are more aware of minimizing risk exposure in time, are:

  1. Determine a key performance indicator which represents the risk reduction benefit of each recommendation.
  2. Determine the optimal order of implementation to maximize risk reduction, while including consideration of minimizing the number of recommendations required.
  3. Improve risk exposure and work efficiency by comparing actual implementation to optimal implementation, and close gaps by analyzing the difference.

By determining the risk reducing benefit of recommendations, simulations can be run which represent what would happen to the risk exposure of potential consequences if the most risk reducing recommendation were implemented. For example, “If Recommendation X was implemented, how would the risk reducing benefit of other recommendations change? How would the risk profile change? What would happen to the risk profile if Recommendation Y were implemented?” Using an iterative methodology, we can determine the optimal order of implementation, and provide an implementation sequence target to strive toward.

  1. Introduction
  2. Recommendation Risk Reduction
  3. Implementing Recommendations (Optimal vs Actual)
  4. Conclusions
  5. References

Process Hazard Analysis (PHA) studies are used, and in some countries required by regulatory bodies, to identify process hazards and operational issues for facilities. These studies enable management, engineers and site personnel to make decisions that reduce risk previously identified in the PHA.

A critical output from most PHA studies is the list of recommendations. These recommendations often are in the form of future safeguards or process design changes that are intended to reduce the risk of consequences. Process facility teams are often expected to implement these recommendations, or address PHA findings and resolve them in a timely manner, soon after the PHA is completed. However, with constraints of time, budget, and resources, and often with company directives or key performance indicators (KPI) which focus on implementing as many recommendations as possible [1], there is a chance for companies to implement recommendations in such a way that can lead to unanticipated risk exposure.

Furthermore, in discussions with site personnel or Process Safety Management (PSM) personnel, 3 key issues are often identified with the current methodology of implementation recommendations:

  • Industry typically uses the KPI “# of Recommendations Implemented”, which incentivizes quantity and not quality of benefit of actions.
  • “Complete all recommendations associated with very high risk scenarios first.” This approach does not consider how risk changes over time as recommendations are implemented.
  • Each layer of an organization has differing understanding of PHA, Risks and Process Safety.

By improving the methodology with which recommendations are prioritized to be implemented, the opportunity for unanticipated risk exposure can be minimized. By taking into account other variables such as the cost, time, and effort it takes to implement a recommendation, the financial and resource expenditures can also be optimized, while having a primary focus on reducing risk.

This paper presents two case studies which demonstrate how integration of PHAs with recommendation optimization analytics can assist in minimizing risk exposure, while potentially optimizing financial opportunity costs. In these case studies, recommendation optimization analytics were performed after PHA recommendations were completely or partially implemented, and the difference between the actual and optimal implementation order was analyzed.

The PHAs analyzed were Hazard and Operability (HAZOP) studies performed on two existing compressor stations. The Center for Chemical Process Safety’s (CCPS) publication on Layers of Protection Analysis (LOPA) was used for the case study when risk reduction credits were applied to existing safeguards and recommendations to be implemented in the future [2].

2.1 Calculating the Risk Reduction provided by Recommendations

A current method often use to assist in prioritizing recommendations is to generate the dataset below (Figure 1.), an export from PHA software tools. In this dataset summary, personnel can utilize the number of scenarios in which a recommendation was applied to, and the maximum residual risk (risk after existing safeguards) associated with a recommendation in order to drive the prioritization process. However, this information is an over summarized dataset that may be misleading because:

  • Personnel have to make several assumptions to determine which recommendation is the most important. For example, although multiple recommendations are shown to be associated with high risk scenarios, personnel are unable to quickly determine how many high risk scenarios each recommendation is related to.
  • Multiple recommendations may be associated with a high risk ranking. However, without observing the severity and likelihoods of the scenarios, it may be difficult to differentiate the benefit of the recommendations.
  • This dataset is a static snapshot in time. Once a recommendation is implemented, the risk profile of scenarios may change and the benefit of remaining recommendations may change.

Figure 1. Sample PHA Recommendation HAZOP Export

In order to minimize these assumptions and more objectively determine the actual risk reduction (benefit) provided by recommendations, the following need to be considered:

  • The current frequency of the consequence and/or the initiating event of the cause.
  • The probability of failure of demand (PFD) of the recommendation of focus and any existing safeguards on related consequences.
  • The severity level and associated tolerable likelihood of each cause/consequence pair the recommendation is associated with. These tolerable likelihoods are usually identified on the Corporate Risk Matrix, and are based on a company’s risk tolerance color zones.

On the presented cases, the following assumptions were made when calculating the risk reduction provided by each recommendation. This was based on the Risk Matrix shown in Figure 2:

  • Initiating Event Frequencies (IEF) of causes were assigned based on the likelihoods provided by the PHA Team. (Table 1.)
  • A PFD of 0.1 was assigned to each safeguard/recommendation given 1 “credit” of risk reduction in the PHA. Safeguards/recommendations given 2 “credits” were assigned a PFD of 0.012.
  • Tolerable Frequencies were determined based on the below assumption for each severity. (Table 2.)

Companies can use specific IEFs and PFDs from their internal equipment databases for the risk reduction calculations to improve the precision of the calculations. Risk Reducing Safeguards and recommendations reviewed in this report are assumed to be specific, independent, dependable and auditable.

Table 1. Frequency Correlation for Risk Matrix Likelihoods

LikelihoodInitiating Event Likelihood (/yr)
10.0001
20.001
30.01
40.1
51

Table 2. Target Tolerable Frequency Likelihoods for Risk Matrix Severities

Severity CodeTolerable Frequency (/yr)
11
20.1
30.01
40.001
50.0001

Figure 2. Case Study Risk Matrix

The methodology used to calculate risk reduction provided by recommendations included the following criteria:

  • Safeguards and recommendations provide only reduction of the likelihood of a consequence from occurring. The severities of the worst credible consequences identified in the PHA were assumed to be constant.
  • The assumed tolerable frequency of a consequence is indicative of its relative impact. The less tolerable a consequence is considered, the more severe its impact.
  • It is assumed that all risk reducing safeguards identified in the PHA are specific, functionally independent from each other and the initiating event, and are dependably effective. If existing safeguards or future recommendations are not as effective as described in the session, the corresponding risk reduction provided for the recommendation may change.
  • The assumed frequency of the cause or initiating event, and the PFD of the safeguards documented can be used to determine the current mitigated frequency of the consequence scenario.
  • The assumed frequency of the cause or initiating event, the PFD of the safeguards, and the PFD of the recommendations being considered as risk reducing (whether they are future safeguards or design changes), can be used to determine the future frequency of the consequence if the recommendation were to be implemented.
  • The risk reduction of a Recommendation is the change in risk associated with implementing that individual recommendation. This means that if multiple recommendations are applied to the same consequence scenario, their collective impact on reducing the risk is not considered as it is not guaranteed that all recommendations on that scenario will be implemented concurrently.
  • Risk reduction across different severities and consequence receptors were aggregated to determine the cumulative risk reduction associated with a recommendation. This means that the more times a recommendation was used in a PHA, the more risk reduction it will provide. These cumulative values can be broken down into different consequence receptors and severities as required.
  • These calculated recommendation risk reductions to be provided are snap shots in time that can be continually updated as recommendations are implemented. This will be further elaborated in Section 3.

2.2 Case Study Recommendation Risk Reduction

Below are “snap shots” for the case study which presents the initially determined static risk reduction of each recommendation identified in the PHAs. Figures 3 and 4 below show:

  • The y-axis represents the recommendation risk reduction provided, and the x-axis shows which recommendation is being analyzed.
  • The data is sorted in descending order of risk reduction provided by recommendations.
  • Each bar, which represents risk reduction, is broken down by risk reduction provided above tolerable risk (green), and risk reduction provided below tolerable risk (blue). For example, if a recommendation reduces risk from a very high risk level to a tolerable risk level, the recommendation bar would only show green. However, if a recommendation reduces risk level from tolerable to low, the recommendation bar would only show blue.
  • Risk reduction across different consequence receptors are aggregated, however can be broken down into each receptor as required.
  • The right y-axis is associated with the bars which represents the recommendation risk reduction. These bars represent the same data as in the prior snapshot charts (Figure 3. and 4.)
  • In Figure 3. (snapshot risk reduction) Rec #25 and #26 had very similar risk reduction, whereas in Figure 5., Rec # 25 had notably less risk reduction. This is because Rec # 25 and # 26 were applied to several of the same consequence scenarios and when Rec # 26 was implemented and reduced the risk of its respective scenarios, Rec # 25 had less risk reduction associated with it.

Figure 3. Recommendation Risk Reduction Snap Shot (Case Study # 1)

In Case Study # 1 (Figure 3.):

  • There is a notable difference between the relative risk reductions provided of the Top 4 recommendations versus the remaining recommendations.
  • Rec #25 and #26 are applied to many of the same consequence scenarios, but are different in risk reduction provided.
  • Rec # 13 and # 15 have slight contributions in criticality from below tolerable risk scenarios. From Rec #24 onwards, all recommendations only have below tolerable risk reduction.

Figure 4. Recommendation Risk Reduction Snap Shot (Case Study # 2)

In Case Study # 2 (Figure 4.):

  • Both Rec #44 and # 45 are tied for risk reduction as they are applied to the same consequence scenarios.
  • Note that Rec # 41, which only has below tolerable risk reduction, is more important than a recommendation which has above tolerable risk reduction. This is primarily because Rec # 41 was utilized many times more than Rec # 50, adding to its cumulative risk reduction.

3.1 Determining the Optimal Order of Implementing Recommendations

As mentioned in the prior sections, simply calculating the risk reduction of the recommendations in a snapshot in time dataset may not be enough to understand how you should sequentially and optimally implement your recommendations.

When decisions are made to implement a recommendation, its impact on the frequency of the consequence and the risk reduction of remaining and unimplemented recommendations needs to be considered. Implementing one recommendation may reduce the importance or the need for sequentially implementing additional recommendations on the same consequence scenario. This iterative methodology will improve the ability of personnel to optimally choose which recommendation should be implemented next, and ensure that risk exposure is minimized. Personnel can also consider financial optimization in scenarios where risk can be reduced to an “as low as reasonably practicable level” (ALARP).

The below Optimal Recommendation Implementation Charts (Figures 5 and 6) identifies the optimal sequential order to implement recommendations to minimize risk exposure through recommendations which provide the most risk reduction. After each recommendation is implemented, calculations are re-run for recommendation risk reduction and consequence risk ranking. In this simulation it is assumed that there are no constraints to implement recommendations such as budget, cost or effort. These parameters can be taken into consideration if provided by those implementing recommendations.

In the charts below:

  • Only risk reducing recommendations are listed.
  • The left y-axis is associated with the stacked chart area, which shows the number of very high and high risk scenarios that remain as recommendations are sequentially implemented. The height of the red section area represents the number of very high risk scenarios, and the height of the orange area section represents the number of high risk scenarios. Note that when the risk ranking of very high risk scenarios are reduced, it is possible that the number of high risk scenarios increase. This is because the risk level of a very high risk scenario can be reduced to high risk through implementation of a recommendation. Depending on the risk matrix, it is also possible that when risk reducing recommendations are implemented, that the risk ranking will not change even though the likelihood of the scenario is reduced.

Figure 5. Optimal Recommendation Implementation by Risk Reduction (Case Study # 1)

In Case Study # 1 (Figure 5.):

  • After all risk reducing recommendations were implemented there are still high risk scenarios remaining. This means that it should be verified whether additional risk reducing recommendations are required in order to achieve the minimum tolerable risk level required across all scenarios. This demonstrates that through this analytic exercise, unintended risk exposure due to sequential recommendation implementation can be quickly identified right after a PHA.
  • Only 4 risk reducing recommendations are required to lower the risk of all high risk scenarios to a tolerable risk level.
  • In Figure 4. Rec # 44 and # 45 had the same criticality, however in Figure 6. Rec # 45 had notably less risk reduction. This is because these recommendations were applied to the same scenarios, and when Rec # 44 was implemented and the risk of the associated scenarios were reduced, the risk reduction provided by Rec # 45 decreased. Without this iterative approach, personnel may think Rec # 45 is still as beneficial as Rec # 44, and have potential financial opportunity cost.

Figure 6. Optimal Recommendation Implementation by Risk Reduction (Case Study # 2)

In Case Study # 2 (Figure 6.):

Across these two optimal sequences (Figure 5. and Figure 6.) it can be seen that:

  • An optimal sequence can help quickly identify whether additional risk reducing recommendations are required in order to achieve tolerable risk level across all scenarios.
  • Implementing a risk reducing recommendation on a consequence scenario will have an impact on the risk exposure and risk reduction of the remaining recommendations identified in that scenario.
  • Although in many cases the recommendation with higher risk reduction will impact more high risk scenarios, it is possible for a recommendation to provide a higher risk reduction and impact fewer high risk scenarios, simply because it was used more times for risk reduction.

3.2 Gap Analysis of the Optimal vs Actual Order of Implementing Risk Reducing Recommendations

The charts below show how recommendations were actually implemented in Case Study # 1 and # 2 without considering an optimal implementation sequence. These charts follow the same format as Figures 5. and 6., except that all recommendations, regardless of whether they are risk reducing or not, are shown in the graph as per the actual implementation results.

Figure 7. Actual Recommendation Implementation (Case Study # 1)

In Case Study # 1 (Figure 7.):

  • It took the team 16 recommendations (~ 6 months from the completion of the HAZOP) to eliminate all very high risk scenarios, whereas optimally (Figure 5.) it would have taken 4 recommendations to achieve a more efficient outcome.
  • In discussion with the key Process Safety contact, the recommendation implementation team was focused on implementing as many recommendations as possible without any specific focus on risk reduction. This meant that they were focused on items which were low effort but also provided low risk reduction.
  • Although there were some critical risk reducing recommendations which could not be completed immediately, others, such as Rec # 13, an additional operator round for operations, could have been implemented much sooner and provided reduced risk exposure.

Figure 8. Actual Recommendation Implementation (Case Study # 2)

In Case Study # 2 (Figure 8.), not all PHA recommendations were implemented at the time of the analysis and there was an opportunity to assist in prioritization the remaining recommendations to attempt to achieve lower risk exposure. Some of these critical recommendations were planned to be implemented in the short term, and others recommendations were not identified in any short term planning. In Case Study # 2.:

  • Recommendations are broken down into three groups:
    • Implemented
    • Planned to Implement
    • Remaining Recommendations
  • Only 4 risk reducing recommendations would have been required to be implemented to reach a tolerable risk level exposure across all scenarios. However, 50% of those recommendations were not in any immediate plans for implementation. This means that there might have been long term unintended risk exposure if this analysis was not performed.
  • Many recommendations were planned to be implemented during an outage, all of which were reducing risk levels from tolerable to below tolerable risk. This is a potential area of budget savings depending on the benefit to cost ratio of these recommendations, and the resources that would potentially have been used could have been utilized instead for the remaining high risk reducing recommendations.

The common issues across these two case studies when comparing Optimal to Actual Recommendation Implementation sequence were:

  • Personnel appeared to be more focused on completing as many recommendations as possible versus the most critical risk reducing recommendations.
  • In both case studies there was a notable difference between the minimum and actual number of recommendations that were implemented before the minimum tolerable risk level would have been reached. This identifies a risk exposure opportunity cost which could have been minimized.
  • In both case studies there were several recommendations with low or no risk reduction and that were implemented before more critical risk reducing recommendations. This identifies the financial and resource opportunity cost that could have been reduced either through postponing or not requiring to implement certain recommendations.
  • Ideally recommendation optimization simulations can be completed immediately after the PHA and provide the implementation team with a sequential guide for implementing recommendations in order to minimize financial cost, and risk exposure.

PHA Recommendation Optimization Analytics can provide a detailed analysis and prioritization assessment of a facility’s PHA recommendations.

Current industry’s methodology for implementing recommendations have key performance indicators with primary focus on implementing as many recommendations as possible, without necessarily focusing on optimally reducing risk exposure.

Based on Case Study # 1 and # 2, it was found that in the process of implementing recommendations for 2 separate PHAs there were situations where the personnel:

  • Unintentionally did not plan to implement recommendations which were associated with providing high risk reduction, and
  • Prioritized recommendations which had little or no risk reduction as they were prioritized based on less required effort to implement.

If recommendations have already been partially or completely implemented from a PHA, recommendation optimization sequencer can be used to determine and find any risk reducing gaps in the implementation sequence, which can be adjusted for future or remaining PHA recommendations.

If no recommendations have been implemented, recommendation optimization can be used to improve awareness of risk exposure and to plan on how to implement a sequence of recommendations, taking into consideration items such as time, effort and cost. This approach can minimize risk exposure and financial and resource opportunity cost.

Link to AIChE site: https://www.aiche.org/conferences/aiche-spring-meeting-and-global-congress-on-process-safety/2018/proceeding/paper/recommendations-and-broken-kpi

  1. AICHE. 2011. Process Safety Leading and Lagging Metrics. [ONLINE] Available at: https://www.aiche.org/sites/default/files/docs/pages/CCPS_ProcessSafety_Lagging_2011_2-24.pdf. [Accessed 20 October 2017].
  2. Center for Chemical Process Safety (CCPS). Guidelines for Enabling Conditions and Conditional Modifiers in Layer of Protection Analysis. Hoboken, NJ, USA: John Wiley & Sons, Inc., 2012
  3. Center for Chemical Process Safety (CCPS). Layer of Protection Analysis. New York, New York, USA. American Institute of Chemical Engineers, 2001
  4. Harvard Business Review. 2017. Know the Difference Between Your Data and Your Metrics. [ONLINE] Available at: https://hbr.org/2013/03/know-the-difference-between-yo. [Accessed 20 October 2017].

You May Also Like…

Loading...