Quality Focused Risk Management Framework for Research and Development Programs Carla Oliveira, Lila L. Carden & Jamison V. Kovach University of Houston Houston Regional Quality Conference November 13, 2015 Houston Texas
Learning Objectives In this session you will: Obtain ideas about how to implement a riskrelated case study Learn about the effectiveness of such methods Gain insights regarding the associated implementation of a Research & Development (R&D) process framework
Agenda Background Justification Methodology Current state map Case study findings Benchmarking Future state suggestions Process correlations Project risk management plan Improved R&D framework Summary Take-aways References
Background Case study Research and Development (R&D) department of a global organization that provides products, systems and services to the oil and gas industry Project focus Analyze the R&D process during six months and develop an improved process framework
Justification Ineffective risk process High uncertainty Unreliable data for strategic decision making Delays/cost overruns
Methodology Map of current state Case-study projects Benchmarking Improved framework proposal Review of records Observation Review of records Literature review PMBOK concepts Interviews Interviews Audit Gap analysis Internal Six Sigma tools
Current State Map Phase 1: Concept evaluation Phase 2: Preengineering Phase 3: Detailed engineering Business case Cost and schedule baselines Risk register Lessons learned TRL assessed WBS FMECA Designs/ drawings Risk register updated FMECA meetings Technology qualification program phases
NASA s Technology Readiness Level (TRL)
Case Study Findings Area Process Finding New releases or updated procedures, policies and templates are not efficient Current processes are based on qualitative assessments Suppliers technology development processes are not addressed by the existing procedures
Case Study Findings Area Training Baselines Finding Some engineers are not skilled in project management techniques Cost and schedule baselines are not developed simultaneously Risks are not linked to schedule Earned Value Management data not used
Case Study Findings Area Risk Assessment Product Risks Finding Only about one third of the risk events identified had assigned owners and action deadlines Risks and opportunities with high impacts were not measured against the business case when the risk response is to accept the risk
Case study findings Area Risk Monitoring and Control Finding Effectiveness of past mitigation actions and residual risks were not assessed No correlations between risk exposure, contingency budget and milestones High impact categories unchanged over time Risk exposure with ninety percent probability of occurrence increased over time
Benchmarking KPIs for Technical Risk Management KPI Red Risks With No Actions Identified Opportunities Overdue Actions Frequency Updates Risks not Assessed Actions Missing Info *Study from Oil and Gas Industry Description (Percent) Open red risks to total number of open risks Open opportunities to total number of open risks Open actions that are overdue to total number of open actions Number of days between assessments Open risks not assessed to the total number of open risks Open actions missing deadlines or responses to the total number of open actions
Benchmarking Automotive Industry FMEA Risk value 344 to 1000 65 to 343 1 to 64 Follow PDCA Cycle until acceptable
Benchmarking Aerospace Industry
Benchmarking PMI Practice Standard for EVM
Benchmarking Construction Industry Risk Performance Index (RPI) Measure the performance in three aspects: cost/schedule/risk Example: Cost Risk Performance Index (CRPI) CRPI = (FCRV-RCRV)/FCRV CRPI: Cost Risk Performance Index FCRV: Forecasted Cost Risk Value RCRV: Residual Cost Risk Value
Future State Suggestions Review Process Use Information for Improvements Update Procedures Monitor Projects / Enforce Model Use Train Users
Future State Suggestions Use software for risk management activities Review technology lifecycle management framework documents Use Earned Value Management (EVM) methods and tools Maintain a risk burn down graph Adopt project KPIs Provide training Perform audits
Process Correlations Six Sigma DMAIC Cycle Define Measure Analyze Improve Control Project Risk Management - PMBOK Plan Risk Management Phase 1 Identify Risks Perform Qualitative/ Qualitative Analysis Phase 2 Phase 3 Plan Risk Responses Control Risks STEP 1: Establish Project Risk Management Plan STEP 2: Identify Technology Failure Modes STEP 3: Assign Likelihood, Impact and Detection Values STEP 4: Calculations Step 5: Visual Tools STEP 6: Risk Response STEP 7: Prioritize STEP 8: Revise RFMEA STEP 9: Technical Performance Review STEP 10: Control Risks
Project Risk Management Plan Scope, Methodology, Definitions, Roles and Responsibilities, References Risk categories Definition of likelihood, impact and detection scales
Scale Examples Scale Likelihood Detection 9-10 Very likely No detection method available or known to alert with enough time to plan for a contingency. 7-8 Will probably occur Detection method effectiveness unknown, unproven or unreliable. 5-6 Equal chance of occurring or not Detection method has medium effectiveness. 3 4 Probably will not occur 1 2 Very unlikely Detection method has moderately high effectiveness. Detection method is highly effective / It is almost certain that the risk will be detected with adequate time.
Scale Examples Scale Schedule Impact Cost Impact Technical Impact 9-10 7-8 5-6 Major milestone impact and >20% impact to critical path Major milestone impact and 10 to 20% impact to critical path 5 to 10% impact to critical path 3 4 <5% impact to critical path Total project cost increase >20% 10 to 20% total project cost increase 5 to 10% total project cost increase <5% total project cost increase 1 2 Insignificant impact Insignificant impact Scope change makes the project output unusable Scope change may compromises functionality Scope change impacts the output and requires client approval. Scope change is minor and may require internal approval Scope changes are minimum
Project Risk Management Plan Risk appetite and tolerance Risk threshold Use the values for risk appetite and risk tolerance to represent graphically the limits in the matrix likelihood versus impact and define zones along the levels of uncertainty and impact
Improved R&D Framework RFMEA Summary Δ TRL = Current TRL Desired TRL ID Date TQ # Root Cause TRC Delta-TRL TNV Failure Mode Effect Phase Technology Index 1 2 n Likelihood/ Impact/ Occurrence Severety Risk Score Detection RPN Risk response Technology Risk Factor = RPN weighted average TRC (API RP 17N) Score A Very High 4 B High 3 C Medium 2 D Low 1 Technology need value Level of development of TQ to system concept 1 Enhancing Qualified alternative solutions could be used with very modest negative impact on the system concept 2 Very significant There may be a work around with very modest negative impact on the system concept 3 Enabling System concept depends on this qualification
Improved R&D Framework Prioritization Tools Summary Bubble chart Pareto diagram Risk Score vs. RPN 12/6/2015 13 Nov 2015 ASQ Houston Regional Conference 26
Summary Classify as EPC Project Follow Standard Risk Procedure NO Perform GAP Analysis TRL <4 YES Classify as TQP (NPD) Perform RFMEA Assess TC/TRC (weights) Define technology failure modes Assess Likelihood, Impact and Detection Calculate individual and total RPN Build graphs Prioritize risks Establish Root Cause/ response Revise RFMEA Monitor RPN
Summary (cont d) Case study achieved its goal, resulting in improved R&D model: More certainty due to better time and budget estimates More effective risk management processes More reliable data for strategic decision making
Learning Takeaway In this session you: Obtained ideas about how to implement a riskrelated case study Learned about the effectiveness of such methods Gained insights regarding the associated implementation of a Research & Development (R&D) process framework
References American Petroleum Institute. (2009). API RECOMMENDED PRACTICE 17N: Recommended Practice for Subsea Production System Reliability and Technical Risk Management. American Petroleum Institute. Bakken, A. T. (2011). Risk management in projects; Monitoring of performance indicators as element in the risk management process. Stavanger: University of Stavanger. DET NORSKE VERITAS. (2013). DNV-RP-A203: Technology Qualification. DET NORSKE VERITAS AS. International Organization for Standardization. (2008). ISO 9001: Quality Management Systems - Requirements. Geneva: International Organization for Standardization. International Organization for Standardization. (2009). ISO 31000: Risk management Principles and guidelines. International Organization for Standardization. Kim, S.-G. (2012). Risk Performance Index and Measurement System. In Z. Haq, Advanced Topics in Measurements (pp. 227-242). InTech.
References (cont d) National Aeronautics and Space Administration. (2014, November 20). Space Communications and Navigation. Retrieved from NASA: http://www.nasa.gov/content/technology-readiness-level/#.vg9odidkori Project Management Institute. (2009). Practice Standard for Project Risk Management. Pennsylvania: Project Management Institute. Project Management Institute. (2013). A Guide to the Project Management Body of Knowledge. Pennsylvania: Project Management Institute, Inc. Sauser, B., Gove, R., Forbes, E., & Ramirez-Marquez, J. E. (2010). Integration maturity metrics: Development of an integration readiness level. Information Knowledge Systems Management, 17-46. Segismundo, A., & Miguel, P. A. (2008). Failure mode and effect analysis (FMEA) in the context of risk management in new product development. A case study in an automotive company. International Journal of Quality & Reliability Management, 899-912. Shenhar, A., Dvir, D., Milosevic, D., Mulenburg, J., & al, e. (2005). Toward a NASA-Specific Project Management Framework. Engineering Management Journal, 8-16.
Thank you for your attention! Questions?