Canada United States Enhanced (CAUSE) Resiliency Series Evaluation Report

Transcription

1 CAN UNCLASSIFIED Canada United States Enhanced (CAUSE) Resiliency Series Evaluation Report International Safety Research Prepared by: International Safety Research 38 Colonnade Road North Ottawa, Ontario Canada K2E 7J6ISR ISR Report Version 3.0 PSPC Contract Number: W /001/SV Technical Authority: Glen Weimer, Project Manager Contractor's date of publication: January 2018 Defence Research and Development Canada Contract Report DRDC-RDDC-2018-C047 March 2018 CAN UNCLASSIFIED

2 CAN UNCLASSIFIED IMPORTANT INFORMATIVE STATEMENTS The information contained herein is proprietary to Her Majesty and is provided to the recipient on the understanding that it will be used for information and evaluation purposes only. Any commercial use including use for manufacture is prohibited. This document was reviewed for Controlled Goods by Defence Research and Development Canada (DRDC) using the Schedule to the Defence Production Act. Disclaimer: This document is not published by the Editorial Office of Defence Research and Development Canada, an agency of the Department of National Defence of Canada but is to be catalogued in the Canadian Defence Information System (CANDIS), the national repository for Defence S&T documents. Her Majesty the Queen in Right of Canada (Department of National Defence) makes no representations or warranties, expressed or implied, of any kind whatsoever, and assumes no liability for the accuracy, reliability, completeness, currency or usefulness of any information, product, process or material included in this document. Nothing in this document should be interpreted as an endorsement for the specific use of any tool, technique or process examined in it. Any reliance on, or use of, any information, product, process or material included in this document is at the sole risk of the person so using it or relying on it. Canada does not assume any liability in respect of any damages or losses arising out of or in connection with the use of, or reliance on, any information, product, process or material included in this document. Template in use: March TEST templatecr EL1.dotm Her Majesty the Queen in Right of Canada (Department of National Defence), 2018 Sa Majesté la Reine en droit du Canada (Ministère de la Défense nationale), 2018 CAN UNCLASSIFIED

3 Defence Research and Development Canada s Centre for Security Science, in partnership with Public Safety Canada

4 Canada-United States Enhanced (CAUSE) Resiliency Series Evaluation Report ISR Report QUALITY ASSURANCE AND VERSION TRACKING Authorization Title Canada-United States Enhanced (CAUSE) Resiliency Series Evaluation Report Document Number Version Version Developed By: Reviewed By: Approved By: 3.0 Approved for Corporate Release By: 1.0 K. Forbes S. Dowker I. Becking M. McCall 17-Mar K. Forbes A. Armitage I. Becking M. McCall 03-Jan-18 Date Jan-18 A. Armitage S. Dowker I. Becking M. McCall Version Tracking Ver. Action By Date 1.0 Release to Client M. McCall 17 Mar Release to Client M. McCall 03 Jan Release to Client M. McCall 16 Jan 2018 Her Majesty the Queen in Right of Canada, as represented by the Minister of National Defence, 2018 Sa Majesté la Reine (en droit du Canada), telle que représentée par le ministre de la Défense nationale, 2018 i

5 Canada-United States Enhanced (CAUSE) Resiliency Series Evaluation Report ISR Report EXECUTIVE SUMMARY Defence Research and Development Canada s Centre for Security Science (DRDC CSS), Public Safety Canada, and the United States (US) Department of Homeland Security (DHS) Science and Technology (S&T) Directorate First Responders Group (FRG) jointly championed, designed, and executed a series of cross-border Canada/US Enhanced (CAUSE) Resiliency experiments. Five CAUSE experiments were completed in this series. For each experiment, a series of planning meetings (initial, mid and final planning) was held in the local geographic regions to ensure that the proposed scenarios and the expected simulated responses represented plausible real-world events. The current study evaluated the impact of the CAUSE experiment series on all participating organizations. A web-based survey (completion response 19%) was developed and administered to previous CAUSE participants in Canada and the US. The participants (N=33) were experienced within the domain of emergency operations and cross-border response and mostly consisted of planning and management personnel. These personnel were largely associated with local Emergency Operations Centres (EOC) and provincial/state government organizations. However, feedback from federal organizations, tri-services, digital volunteer organizations and not-for-profit organizations were represented in the data set. The survey, comprised of both closed-ended and open-ended questions, was first administered November, 2016 to personnel who participated in the CAUSE I through IV experiments. Subsequently, a subset of the survey questions were posed to CAUSE V participants upon completion of the fifth experiment in November For evaluation purposes, the main technologies used in the CAUSE experiments were categorized according to their primary function within the experiments. Impacts (i.e., support to people, process and technology) of participating in the CAUSE experiment series were also evaluated through this study. The data were analyzed using descriptive statistics. Feedback related to the overall CAUSE experiment series as well as the specific CAUSE experiments was gathered. These findings provided insight into the organizational use of technology during day-to-day and emergency operations, evaluated the progress that has been made with respect to implementing the lessons identified from many of the experiments and identified some of the remaining challenges related to cross-border emergency operations. Taken together, these findings have identified the positive impacts and challenges associated with the CAUSE experiment series on the participating organizations. The study findings will inform the governments with regard to the impact of the CAUSE experiment series on cross-border emergency management response. These findings can be used to help inform the development of future cross-border experiments and exercises. ii

6 Canada-United States Enhanced (CAUSE) Resiliency Series Evaluation Report ISR Report TABLE OF CONTENTS Executive Summary... ii 1. Introduction Background CAUSE Series and BTB CAUSE Experiments Document objective Document scope Overview of the report Methodology CAUSE series evaluation Participants Survey Design Survey Administration Results Organizational Roles Type of organization Participants involvement in CAUSE experiments Summary of CAUSE I feedback Summary of CAUSE II feedback Summary of CAUSE III feedback Western experiment Northeastern experiment Summary of CAUSE IV feedback Vignette 1: Canadian-led patient transfer Vignette 2: US-led social media response Summary of CAUSE V feedback Impacts of CAUSE technology on real-world scenarios Impacts of CAUSE technology on people, processes and technology People Processes Technology Improving emergency operations through CAUSE Remaining challenges for cross-border operations Summary LIST OF FIGURES Figure 3-1: Organizational roles Figure 3-2: Type of organization Figure 3-3: Respondents' involvement in CAUSE experiments Figure 3-4: Usefulness of CAUSE I technology in day-to-day operations Figure 3-5: Usefulness of CAUSE I technology in emergency operations Figure 3-6: Implementation of lessons identified during CAUSE I experiment Figure 3-7: Usefulness of CAUSE II technology in day-to-day operations Figure 3-8: Usefulness of CAUSE II technology in emergency operations iii

7 Canada-United States Enhanced (CAUSE) Resiliency Series Evaluation Report ISR Report Figure 3-9: Implementation of lessons identified during CAUSE II experiment Figure 3-10: Usefulness of CAUSE III Western technology in day-to-day operations Figure 3-11: Usefulness of CAUSE III Western technology in emergency operations Figure 3-12: Implementation of lessons identified during CAUSE III Western experiment Figure 3-13: Usefulness of CAUSE III Northeastern technology in day-to-day operations Figure 3-14: Usefulness of CAUSE III Western technology in emergency operations Figure 3-15: Implementation of lessons identified during CAUSE III Northeastern experiment Figure 3-16: Usefulness of CAUSE IV Vignette 1 technology in day-to-day operations 31 Figure 3-17: Usefulness of CAUSE IV Vignette 1 technology in emergency operations 32 Figure 3-18: Implementation of lessons identified during CAUSE IV Vignette I experiment Figure 3-19: Usefulness of CAUSE IV Vignette 2 technology in day-to-day operations 34 Figure 3-20: Usefulness of CAUSE IV Vignette 2 technology in emergency operations 35 Figure 3-21: Implementation of lessons identified during CAUSE IV Vignette 2 experiment Figure 3-22: Impact of CAUSE V technology / application on emergency operations Figure 3-23: Short-term impacts of CAUSE technology on emergency operations Figure 3-24: Impact of CAUSE technology on efficiency Figure 3-25: Impact of CAUSE technology on decision-making support Figure 3-26: Impact of CAUSE technologies on information sharing Figure 3-27: Overall impact of CAUSE on people Figure 3-28: Overall response to processes explored during CAUSE Figure 3-29: Overall response to technology explored during CAUSE Figure 3-30: Impacts on improving emergency operations Figure 3-31: Remaining challenges to cross-border emergency operations iv

8 Canada-United States Enhanced (CAUSE) Resiliency Series Evaluation Report ISR Report INTRODUCTION 1.1 Background CAUSE Series and BTB On December 7, 2011, President Obama and Prime Minister Harper released the Beyond the Border (BTB) Action Plan [1]. This policy exists at a bi-national level between Canada and the United States (US). The BTB Action Plan outlines an objective to rapidly respond to and recover from disasters and emergencies on either side of the border by establishing bi-national plans and capabilities for emergency management. This plan has a focus on communications interoperability and chemical, biological, radiological, nuclear, and explosives (CBRNE) events. The BTB Action Plan established two cross-border working groups, including the Canada-US (CANUS) CBRNE Working Group and the CANUS Communications Interoperability Working Group (CIWG). In 2012, the CANUS CIWG developed a five-year work plan with specific goals and activities, several of which are addressed through the CAUSE series. To address the intent of the BTB Action Plan, Defence Research and Development Canada s Centre for Security Science (DRDC CSS), Public Safety Canada (PS), and the United States (US) Department of Homeland Security Science and Technology Directorate First Responders Group (US DHS S&T FRG) jointly championed, designed, and executed a series of cross-border Canada-US Enhanced (CAUSE) Resiliency experiments. The CAUSE experiment series is focused primarily on cross-border technology integration with the potential to enhance Situational Awareness (SA) for Emergency Management (EM) organizations. The overarching hypothesis for this series of experiments is that leveraging interoperable technologically to enhance cross-border SA, measurably improves regional resilience. The detailed objectives of the CAUSE Resiliency experiment series are listed below: Connect, test and demonstrate emerging operational technologies; Advance EM and responder SA capabilities; Demonstrate the value of federal S&T investments; Demonstrate enhanced resilience through improved interoperable shared SA and mutual aid during major events; Enhance resilience in border regions by leaving behind working operational interfaces, processes, training and exercises that will improve shared SA; and Execute CAUSE as a catalyst to build trust relationships in support of the BTB Action Plan CAUSE Experiments Five scenario-based experiments have been conducted in Canadian and US crossborder communities from 2011 to These experiments were jointly sponsored by DRDC CSS and DHS S&T. A series of planning meetings (Initial Planning Conference (IPC), Mid-Planning Conference (MPC) and Final Planning Conference (FPC)) were held in advance of conducting each experiment. These meetings were held in the local geographic regions where the experiments were planned to occur. Representatives from the local EM agencies attended these meetings to ensure that the proposed scenarios and the expected simulated responses represented plausible real-world 5

9 Canada-United States Enhanced (CAUSE) Resiliency Series Evaluation Report ISR Report events within these geographic regions. A combination of existing and emerging technologies was used for each experiment to simulate the intended cross-border emergency response for these experiments. Government and private industry organizations provided access to technologies and subject matter expertise during the experiment. In general, the emerging technologies were used to demonstrate an alternate means through which the simulated real-world emergencies can be addressed. With the exception of one experiment (CAUSE III Western) the experiments were not focussed on making direct comparisons between the existing and emerging technologies but rather to test the new capabilities enabled by the emerging technologies. A detailed evaluation was completed during, and upon conclusion of each of the CAUSE experiments. Each of these experiments is briefly described below and some of the relevant reports are referenced. CAUSE I (June 2011) was held in British Columbia (BC), Canada and Washington State (WA), US. This experiment was focussed on enhancing systems interoperability between Canada and the US. A major earthquake scenario (Cascadian subduction zone earthquake off the coast of Oregon of magnitude 9.0) was developed which provided the context for the cross-border emergency response [2]; CAUSE II (March 2013) was held in New Brunswick (NB), Canada and Maine (ME), US. This experiment was focused on information exchange between local, state, provincial and national organizations using various systems, agencies and software applications. Two scenarios were developed which provided the context for the cross-border emergency responses. On the first day, the scenario was based in NB and was concerned with the response to an industrial explosion that was scripted to have occurred at a private industry location. On the second day, the scenario was based in ME and was concerned with a major highway collision involving a truck hauling hazardous material [3][4]; CAUSE III (November 2014) was held in two locations. Two scenarios were developed which provided the context for the cross-border emergency responses. The first scenario (Northeastern) was focussed on enhancing SA using interoperable tools and digital volunteers to support emergency operations. [5] The scenario addressed the recovery phase of a large hurricane which was scripted to occur along the northeastern coast of both nations. The experiment was based in Nova Scotia (NS) Canada and in two locations in the US including New Hampshire (NH) and ME. The second scenario (Western) was focussed on augmenting a Land Mobile Radio (LMR) system with a deployable 700MHz Public Safety Broadband Network (PSBN) using Long Term Evolution (LTE). The scenario provided the context for a multi-agency emergency response to a wild land fire; the scenario was based near the Canada-US border (Willow Creek Port of Entry) between and Saskatchewan (SK), Canada and Montana (MT), US as well as near the inter-provincial border between Alberta (AB) and SK [6]; CAUSE IV (April 2016) was held in Ontario (ON), Canada and Michigan (MI), US. A tornado scenario that affected both nations was scripted to occur within this 6

10 Canada-United States Enhanced (CAUSE) Resiliency Series Evaluation Report ISR Report region. Two separate vignettes were developed to address unique aspects of an emergency response. The first vignette was focussed on establishing an interoperable 700MHz PSBN to enable communications during cross-border patient transfer. The second vignette was focussed on enhancing SA during the emergency response by using digital volunteers, crowd-sourcing and crossborder public alerting [7][8]. CAUSE V (November 2017) was held in the lower mainland of British Columbia (BC), Canada and the northwest region of Washington State (WA), US. A flood scenario, aggravated by lahars following an eruption of Mt. Baker, was used as the basis for the experiment. The experiment investigated the use of Long-Term Evolution (LTE) technology and real-time data to create a cross-border common operating picture (COP) during the planning, response and recovery phases of emergency operations. As well, the inclusion of digital volunteers and the processes for moving specialized resources and personnel across the Canada- US border were evaluated, leveraging the use of the PSBN and interoperable applications. Finally, CAUSE V explored the use of the Pacific Northwest Emergency Management Arrangement (PNEMA) for state-to-provincial mutual aid requests [9]. 1.2 Document objective This report documents the outcomes of the evaluation that was performed on the five experiments included within the CAUSE series. 1.3 Document scope This report documents the design, methodology and findings from the CAUSE series evaluation. 1.4 Overview of the report This report is organized into the following sections: Section 1: Introduction This section presents the project background and objectives. It also provides an overview (this section) of the material presented in the report; Section 2: Methodology This section presents the approach used to develop the research design; Section 3: Results This section presents the survey findings gathered using the online survey, augmented with findings from the interviews where applicable; Section 4: Summary This section presents a summary of the key findings; Annex A: Online survey 7

11 Canada-United States Enhanced (CAUSE) Resiliency Series Evaluation Report ISR Report This Annex presents the online survey questions; and Annex B: invitation This Annex presents an example of the invitation that was sent to participants. 8

12 Canada-United States Enhanced (CAUSE) Resiliency Series Evaluation Report ISR Report METHODOLOGY 2.1 CAUSE series evaluation DRDC CSS, in collaboration with DHS S&T FRG, evaluated the impact of the CAUSE experiments on the participating organizations. For the purposes of the current evaluation, the technologies were categorized, according to their primary function during the experiments, by the study team. This approach was used to gather feedback about the main types of technologies that have been explored during the CAUSE experiment series. A web-based survey was developed to gather feedback from previous CAUSE participants. In some cases, the findings were augmented with data gathered via phonebased interviews with previous participants. The evaluation was focussed on identifying the impacts, both short-term and long-term, of the CAUSE experiments on the participating organizations and the challenges that remain within these organizations. 2.2 Participants The study participants (N=33) had all participated in one of the five CAUSE experiments. Prior to the conduct of CAUSE V, participants associated with the previous experiments were invited to complete this web-based survey. Upon completion of the CAUSE V experiment in November 2017, CAUSE V participants were invited to complete a subset of the survey questions. An initial invitation and subsequent reminder s were sent to approximately 174 personnel (approximately 74 participants from the US and 100 participants from Canada). Participants were identified from records that had been maintained during the conduct of the previous CAUSE experiments. It is important to note that some of the participants were no longer in their positions and could not be contacted. The survey response rate associated with the CAUSE evaluation survey was approximately 19%. In addition, some participants preferred to be interviewed by the study team rather than complete the web-based survey. 2.3 Survey Design An online survey was generated by the study team to gather feedback from the participants. Some of the survey questions were relevant to all participants and some survey questions were specific to the CAUSE experiment in which participants were involved. The responses provided by the participants determined the subsequent questions that were presented to them. The complete set of survey questions presented to participants from CAUSE experiments I-IV and CAUSE V are presented in Annex A. The surveys contained both closed and open-ended items. The closed-ended questions required the selection of a single response, to route questions or gather specific information. In addition, participants were able to make multiple selections or provide ratings using a specific scale when closed-ended questions were presented. In contrast, the open-ended questions provided the participants with the opportunity to explain their opinions. The open-ended items were used to identify the benefits 9

13 Canada-United States Enhanced (CAUSE) Resiliency Series Evaluation Report ISR Report associated with the lessons that were implemented by their organizations and the process or procedure that has had the greatest impact on improving emergency management operations. The information used to generate these survey questions was identified from the evaluation reports completed following the completion of each CAUSE experiment. Finally, an open-ended question was presented to participants from CAUASE I-IV to gather information related to the challenges that remained with regard to cross-border emergency operations that were considered when planning CAUSE V. A Welcome Page described the intent of the study and was followed by instructions which provided information needed to complete the survey and contained contact information for the designated point of contact if the participants required any assistance. Subsequently, participants were presented with questions which addressed a range of aspects related to the short-term and long-term impacts for their organizations from participating in the CAUSE experiment series. Additionally, questions were presented which addressed outstanding challenges related to emergency management operations. A Thank-you Page, which again contained the contact information for the designated point of contact, was presented as the final page in the survey. The survey data were analyzed using descriptive statistics. Means and standard deviations were calculated either through Excel or the built-in analysis functions of Survey Monkey. Standard deviations were calculated based on the Population Standard Deviation Formula. [10] The entire dataset and summaries of the data were downloaded for analysis by the study team. The data were subjected to an initial screening process to identify missing data and to calculate the number of responses that were received for each item. The sample size for each item is reported to address situations where missing data points were identified. 2.4 Survey Administration The survey was available in English. Contact information was provided at the beginning of the survey and at the end of the survey in the event that participants had questions or comments. The survey was built and administered using A survey could be completed once from each internet-enabled device. The initial survey which gathered feedback from CAUSE I-IV participants was available during a two-week period between November, Subsequently, the participants involved in CAUSE V were able to complete the survey during a one-week period between 27 November 8 December, Participants from CAUSE I - IV were invited to participate in a guided survey completion session to facilitate the real-time completion of the survey (Annex B). Three Webex meetings were scheduled and led by the study team during which participants were provided with the survey questions and walked through each question as they completed the survey. This support was provided by the study team due to the length of time that had passed since some of the earlier CAUSE experiments had been conducted. In addition, the study team was available to clarify any specific questions regarding the way(s) that the technology was used during the experiments. Based on feedback provided to the study team during these support sessions, the participants were better equipped to explain the impact of the technologies on their organizations. Participants from CAUSE V did not require this additional support since the survey was administered 10

14 Canada-United States Enhanced (CAUSE) Resiliency Series Evaluation Report ISR Report within two weeks of completing the CAUSE V experiment. 11

15 Canada-United States Enhanced (CAUSE) Resiliency Series Evaluation Report ISR Report RESULTS This section presents the results of the survey and the phone-based interviews that were completed by the study team. It is important to note that some questions were presented to participants based on their previous answers (e.g., their involvement in a specific CAUSE experiment, their organization, etc.). Further, not all questions required participants to enter responses such that some participants skipped non-required questions. Therefore, not all of the questions were presented to all of the participants. The number of participant responses that were gathered for each question is noted in the sub-sections below. The survey data were summarized using descriptive statistics in order to describe the patterns associated with the observed responses. It is important to note that in instances where the standard deviation is higher than 1.0, there was substantial variability in the results. This variability was exacerbated due to the small sample size of the collected data. The findings from these analyses are presented using percent frequency and are rounded to the nearest whole percentage. For this reason, the percentages reported for individual questions may not sum to 100 percent. 3.1 Organizational Roles All participants were asked to select the role that they currently have within their organization (N=33). Approximately half of the participants were in management roles (52%) within their organizations (Figure 3-1). Approximately, one-quarter of participants were in planning roles (27%). Substantially smaller percentages of participants were associated with response roles (9%), technical support roles (6%), volunteer organizations (3%) and public relations organizations (3%). See Figure 3-1. It should be noted that since more than half of the participants who completed the survey are experienced emergency managers, it is expected that the data described in this report is heavily weighted in favour of this role. This has the potential to bias the results in favour of the opinions of the emergency managers. However, due to the small sample size of this data, there are many instances where other roles have a near equal weight within a specific experiment. Both of these cases are limitations of the analysis. 12

16

17

18

19

20

21

22

23

24

25 Canada-United States Enhanced (CAUSE) Resiliency Series Evaluation Report ISR Report information. 3.6 Summary of CAUSE III feedback Five participants indicated they were involved with the CAUSE III experiment (See Section 3.3); this accounted for 15% of the participants in the data sample. One of these participants represented a member of a Virtual Operations Support Team (VOST), based in the US, and was involved in both scenarios. The findings related to the Western and Northeastern experiments are presented separately in the sub-sections directly below Western experiment Three of the CAUSE III participants were associated with the Western experiment. One of these participants was associated with a Canadian provincial organization (Alberta First Responder Radio Project (AFRRCS) Program) and the other two were associated with US organizations including a state organization (Hill County (Montana) Sheriff s Department) and a US VOST. The participant associated with the VOST was involved in both CAUSE III scenarios (i.e., Western and Northeastern scenarios). The tools that were used in the CAUSE III Western experiment were categorized into the following three categories: Land Mobile Radio (LMR), deployed Public Safety Broadband Network (PSBN) and software tools over PSBN (e.g., , Skype, Multi- Agency Situation Awareness System (MASAS)). Participants were asked to rate the extent to which these types of tools were useful during day-to-day operations using a language ladder that was treated as a 5-point scale; a 1 indicated the technology or software was Not at all useful whereas a 5 indicated the technology or software was Extremely useful. A rating of 3 indicated that the technology or software was Useful. The results indicated that the LMR (M=4.0, sd=0.8) was indicated to be Very useful during day-to-day operations (Figure 3-10). Further information gathered from the participants indicated that the LMR was used to coordinate cross-border response efforts. In contrast, the deployed PSBN (M=2.5, sd=0.5) and associated software tools that can be used over the PSBN (M=2.5, sd=0.5) were indicated to be only somewhat useful during day-to-day operations. The rating associated with the deployable PSBN was expected since this system is intended, and potentially will only be available even when implemented, to be deployed during an emergency. However, higher ratings associated with using common software tools were expected for day-to-day operations because they are typical office-based tools. It is possible that the participating organizations do not use these tools because of their reliance on the voice-based LMR capability. 22

26

27

28

29

30

31

32

33 Canada-United States Enhanced (CAUSE) Resiliency Series Evaluation Report ISR Report organizations. This variability led to a follow-up question which asked them to identify any benefits associated with proceeding with the implementation of the lessons that were identified during CAUSE III Northeastern experiment. The participants indicated that there was substantial benefit to developing a CONOPs for covering the VOST operations to support conditions of use during a cross-border incident. In addition, there was also a benefit to aligning local, state, national and bi-national approaches for resource management. In contrast, implementing the other lessons (i.e., refining the roles and responsibilities of DVs and exchange of standards-based information) was associated with a minimal benefit. 3.7 Summary of CAUSE IV feedback Twelve participants (36%) indicated they were involved with the CAUSE IV experiment (See Section 3.3). The findings related to Vignette 1 and Vignette 2 are presented separately in the sub-sections directly below Vignette 1: Canadian-led patient transfer Two of the CAUSE IV participants were associated with Vignette 1. These participants were associated with the Canadian-led vignette which simulated cross-border patient transfer. Both of these participants were associated with Canadian organizations (Lambton County Emergency Medical Services and the Federal Bridge Corporation). The tools that were used in CAUSE IV Vignette 1 were categorized into the following two categories: PSBN and software tools leveraged through the use of the PSBN. Participants were asked to rate the extent to which these types of tools could be useful during day-to-day operations using a language ladder that was treated as a 5-point scale; a 1 indicated the technology or software was Not at all useful whereas, a 5 indicated the technology or software was Extremely useful. A rating of 3 indicated that the technology or software was Useful. The results indicated that both types of tools were indicated to be Extremely useful during day-to-day operations (Figure 3-16). It is noted that the PSBN tools are not currently available for day-to-day operations, however if this capability was available, then it seems that the participants would find it to be extremely useful because it would allow their organizations to maintain communications between paramedics and dispatch (and other organizations such as border agencies and the bridge authorities as needed) during normal cross-border patient transfers. 30

34

35

36

37

38

39 Canada-United States Enhanced (CAUSE) Resiliency Series Evaluation Report ISR Report Visualize social media data. This would include modifying the systems used to collect, streamline and visualize the large quantity of response data obtained through social media, crowd-sourcing, damage assessments, digital volunteers and operators (M=3.4, sd=1.1); and Target social media recipients. This would include establishing pre-identified notification groups to ensure all official emergency alerts target the relevant officials and citizens (M=3.4, sd=1.4). In contrast, participants indicated that little progress had been made with respect to implementing the remaining three lessons. These lessons are identified and described below. Joint training for Emergency Operations Centres and DVs. This would include implementing joint training between EOC staff and digital volunteers to strengthen the understanding of the role of social media in support of emergency management (EM) (M=2.8, sd=1.1); Improve official broadcast alert and warning systems. This would include integrating binational-level alert and warning systems, including the U.S. Integrated Public Alert and Warning System (IPAWS) and Canadian National Public Alerting System, in order to improve the efficiency of official-to-official broadcast alerts (M=2.2, sd=1.3); and Establish cross-border Mutual Aid Agreements (MAAs). This would include developing a system to allow local officials to have the authority to enter into cross-border mutual aid agreements (M=2.5, sd=1.2). Similarly to the other experiments, there were substantial differences in the participants opinions as to if the lessons identified had been implemented within their emergency operations. In nearly each situation, the participants are divided in their opinion of this matter. This is an indication that the lessons have been implemented at varying degrees, depending on the organization involved. 36

40

41 Canada-United States Enhanced (CAUSE) Resiliency Series Evaluation Report ISR Report Summary of CAUSE V feedback Eleven participants indicated they were involved with the CAUSE V experiment (See Section 3.3). This accounted for 33% of the participants in the data sample. The study team administered the CAUSE Series Survey to CAUSE V participants within two weeks of completing the CAUSE V experiment. Since the CAUSE V participants had recently contributed a great deal of feedback through the CAUSE V evaluation process, the study team minimized the data collection requirements for this survey. Further, the short timeframe imposed limitations on the use of the technology in day-to-day and emergency operations and the implementation of any lessons learned as a result of participating in CAUSE V. Consequently, the study team selected a sub-set of survey questions from the CAUSE Series Survey. Most of these findings are reported in the sub-sections below which are concerned with the impacts of participating in the CAUSE series. However, a brief summary of the specific feedback related to CAUSE V is presented below. The CAUSE V participants who completed the survey were associated with both Canadian (CBSA, City of Abbotsford, Langley Emergency Program) and US (CBP, Whatcom County Sheriff s Office Division of Emergency Management, Washington State Emergency Management Division) organizations. Most of the participants were associated with a US organization (70%). Some participants did not identify the organization with which they were associated. The CAUSE V participants were asked to identify, based on their observations, which of the technologies used in the experiment had the greatest impact on emergency operations. The results suggested that deployed PSBN (30%), software used over the PSBN (20%) and social media (20%) had the greatest impact on emergency operations. To a lesser extent, participants indicated that GIS (e.g., ArcGIS Online) and supporting applications (e.g., damage assessment, planning and tracking tools such as Survey123, Drakontas Dragonforce, TRX) had an impact on emergency operations (Figure 3-22). All participants rated the above technologies as having a greater impact on emergency operations than emergency operations management tools (e.g., Mutual Aid Resource Planning, Common Operating Picture). One participant indicated that they had reservations with completing this question since a direct comparison with existing technologies was not being performed. 38

42

43

44

45

46

47

48

49

50

51

52 Canada-United States Enhanced (CAUSE) Resiliency Series Evaluation Report ISR Report SUMMARY The current study evaluated the impact of the CAUSE experiments on the participating organizations and the challenges that remain within these organizations. The evaluation was not focussed on making direct comparisons between the existing and emerging technologies but rather to evaluate alternate means through which the simulated realworld emergencies were addressed. A web-based survey was administered to previous participants in the CAUSE experiment series. The survey response represented approximately 19% of the invited audience. This response yielded a very small sample (N=33) but did include representation from both Canada and the US for each of the five CAUSE experiments. In addition, the study team conducted phone-based interviews with a few participants in lieu of completing the survey. The small sample size yielded many complications and limitations within the analysis. It is recognized that a larger sample size would have afforded insights from additional groups of participants and offered other perspectives into the use of the technology and the implementation of lessons learned. Most of the responses were obtained from planning and management personnel who are associated with EOCs and state/provincial government organizations. Therefore, these participants are experienced EM professionals and are informed about the types of real-world scenarios that exist within their geographic regions. As well, approximately two-thirds of the responses were obtained from personnel who had participated in the most recent CAUSE experiments (CAUSE IV and CAUSE V). This evaluation addressed the impacts of the overall CAUSE experiment series on participating organizations. Based on the qualitative feedback, the greatest impacts of participating in the experiments were enhancing the processes associated with real-time communications (i.e., notifications, alerts and warnings), sharing / gathering information, visualizing data during emergency operations that can be shared (i.e., mapping and displaying data) and enhancements to SA. The findings indicated that improvements to efficiency, response coordination (including the development of a COP), satisfying information requirements and supporting decision-making were observed. However, there was some variability in the responses as some participants indicated that decision-making and SA was not enhanced within their organizations. Further, some participants indicated that the technology did not appear to support the clarification of the roles and responsibilities within and between organizations, enhance the use of volunteers during emergency operators or make information more accessible to the public. Overall, this feedback suggests that the technology has some positive impacts however, it does not (and was not intended) to replace the current systems that are in use by these organizations. The findings also suggest that positive changes to the knowledge and experience of people and the evolution of processes and technology can be observed. In general, participating in the CAUSE experiment series supported personnel in performing their roles during real-world incidents. Similarly, participants generally agreed that the processes and technologies explored during the CAUSE experiments were effective (with some notable exceptions related to cross-border response) when dealing with 49

53 Canada-United States Enhanced (CAUSE) Resiliency Series Evaluation Report ISR Report emergency operations. It is recognized that the technologies explored during the CAUSE experiment series support (rather than replace) the existing processes and technologies that are used by the participating organizations. Also, while the Federal governments ensured there were some technology-related leave-behinds, these tended to be software-based tools rather than equipment. In some cases, the implementation of the capabilities was not possible because the equipment could not be left behind (i.e., PSBN). Feedback related to the specific CAUSE experiments was gathered. With the exception of CAUSE V which was completed two weeks prior to the conduct of this survey, the findings revealed that the types of technologies were considered to be Very useful or Extremely useful in day-to-day operations tend to be used more extensively during emergency operations (i.e., mobile communications, GIS tools for SA, LMR, social media). As well, some types of technology that are considered to be Useful during dayto-day operations tend to used more frequently during emergency operations. This pattern of response was expected because emergency operations require an escalation of response. The observation that the technologies explored in the CAUSE experiment series supported (but did not replace) this escalated response was considered a positive impact of the CAUSE experiments. The findings indicated that many of the lessons that were identified following the completion of the CAUSE experiments had been implemented. [2][3][4][5][6][7][8] It is noted that the timeframe between the completion of CAUSE V and the conduct of this survey was too short to allow organizations to implement lessons that arose from the experiment. Therefore, these types of questions were not presented to these participants. In previous experiments, in cases where participants identified a benefit for implementing lessons, these were either already implemented or were still in the process of being implemented. In these cases, barriers for implementation were typically associated with inadequate time to complete the implementation. Other barriers included the need for establishing appropriate legislation to facilitate cross-border response coordination, cost, and the availability of people and equipment. Taken together, these findings have identified the positive impacts, some limitations and challenges associated with the CAUSE experiment series on the participating organizations. The output of this evaluation will be used to identify the impact that the CAUSE experiment series has had on the cross-border emergency management (EM) community since its inception. 50

54 Canada-United States Enhanced (CAUSE) Resiliency Series Evaluation Report ISR Report REFERENCES [1] United States Canada Beyond the Border: A shared vision for perimeter security and economic competitiveness [Online]. Available: Accessed: Mar. 12, [2] Galbraith, J., M. Miller and G. Li. October CAUSE Resiliency (West Coast) Experiment Final Report, DRDC CSS CR [Online]. Available: Accessed: Mar. 12, [3] Vallerand, A., D. Boyd et. al. July Canada-U.S. Enhanced Resiliency Experiment Series CAUSE Resiliency : A second Canada-U.S. Resiliency experiment (CAUSE Resiliency II). [4] Forbes, K. (2013). Public Security Technical Program Task 15 Support to CAUSE Resiliency II Experiment. Technical report to the Centre for Security Science. [5] Dawe, P., Cotter, D. et al. June Canada-U.S. Enhanced Resiliency Experiment Series CAUSE III : Northeastern experiment after action report, DRDC- RDDC-2015-N096. [Online]. Available: Accessed: Mar. 12, [6] Forbes, K. & Becking, I. (2015). Canada-U.S. Enhanced (CAUSE) Resiliency III Western Scenario After-Event Report, Technical report to the Centre for Security Science. [7] Dawe, P., Cotter, D. et al. June Canada-U.S. Enhanced Resiliency Experiment CAUSE IV Binational After Action Report. [Online]. Available: pdf [8] Forbes, K. and Armitage, A. (2016). Canada-US Enhanced (CAUSE) IV Resiliency Experiment Data Report. Technical report to the Centre for Security Science. [9] Dowker, S. (2017). CAUSE V Experiment Plan. Technical report to the Centre for Security Science. [10] SurveyMonkey. Help Center Basic Statistics. Available: [11] Armitage, A. (2017). CAUSE V Evaluation Framework. Technical report to the Centre for Security Science. 51

55 Canada-United States Enhanced (CAUSE) Resiliency Series Evaluation Report ISR Report ANNEX A. SURVEY QUESTIONS A.1 CAUSE I-IV SURVEY This survey has been included as a separate PDF document titled "CAUSE Evaluation survey questions". A.2 CAUSE V SURVEY This survey has been included as a separate PDF document titled "CAUSE V Evaluation survey questions". 52

56 Canada-United States Enhanced (CAUSE) Resiliency Series Evaluation Report ISR Report ANNEX B. INVITATION Hello, The Defence Research and Development Canada's Centre for Security Science (DRDC CSS) and the United States (U.S.) Department of Homeland Security Science & Technology Directorate (DHS S&T) would like to invite you to participate in our Canada- U.S. Enhanced Resiliency Experiment (CAUSE) Series Participant Feedback Questionnaire. The CAUSE series is a series of bi-national experiments focused on improving the interoperability of communications and shared situational awareness between responders using new and emerging technologies. This CAUSE questionnaire will gather information about your experience with the technologies, software, processes and procedures that were included and implemented during the CAUSE series. The information gathered will support the development of future CAUSE experiments by identifying the most frequently used technologies during day-to-day and emergency operations, technologies that can enhance emergency response, existing challenges and lessons learned related to the adoption of the technology and future areas of focus that will improve interoperability in emergency response in border regions. Each survey contains approximately 21 questions and the survey respondent can expect to spend approximately minutes completing the survey. The questionnaire will be available for completion between November 15 and November 30, A series of completion sessions have been scheduled to assist respondents in the completion of the survey. These sessions will take place through WebEx with experienced facilitators with in-depth knowledge of the CAUSE processes and technology. The surveys are scheduled at the following times: November 17, 2016 at 2:00pm EST; November 22, 2016 at 1:00pm EST; and November 28, 2016 at 10:00am EST. Please be sure to respond to NAME ( ) by November 15th with your confirmation in completing this survey and your choice in survey completion session. Once a response is received, a formal meeting invitation for this session will be sent. Questions, comments and concerns can be directed to NAME ( ). Thank you, your participation will provide valuable input to the CAUSE experiment series. 53

57 Canada-United States Enhanced (CAUSE) Resiliency Series Evaluation Report ISR Report ANNEX C. AB AFRRCS BC BTB CAP CANUS CAUSE CBRNE CIWG COP CP CONOPS DHS S&T FRG DRDC CSS DV EDT EEI EOC EM EMBC EMS FPC GIS IM IPAWS IPC LE LMR LTE MAA MASAS ME MEMA MI MOU MPC MT NB NB EMO NH NIEM NS PSBN ON SA ACRONYMS Alberta Alberta First Responder Radio Project British Columbia Beyond the Border Common Alerting Protocol Canada-US Canada-US Enhanced Resiliency Experiment Chemical, Biological, Radiological, Nuclear, Explosives Communications Interoperability Working Group Common Operating Picture Canadian Profile Concept of Operations Department of Homeland Security Science and Technology Directorate, First Responders Group Defence Research and Development Canada Centre for Security Sciences Digital Volunteer Exercise Design Team Essential Element of Information Emergency Operations Centre Emergency Management Emergency Management British Columbia Emergency Medical Service Final Planning Conference Geographic Information System Incident Management Integrated Public Alert and Warning System Initial Planning Conference Law Enforcement Land Mobile Radio Long-Term Evolution Mutual Aid Agreement Multi Agency Situation Awareness System Maine Maine Emergency Management Agency Michigan Memorandum of Understanding Mid Planning Conference Montana New Brunswick New Brunswick Emergency Measures Organization New Hampshire National Information Exchange Model Nova Scotia Public Safety Broadband Network Ontario Situation Awareness 54