D7.1 : Validation plan formation and report

Transcription

1 This project has received funding from the European Union s Seventh Framework Programme for research, technological development and demonstration under grant agreement n Project Acronym: CRe-AM Project Full Title: Creativity REsearch Adaptive roadmap Grant Agreement: Project Duration: 24 months (Oct Sept. 2015) D7.1 : Validation plan formation and report Authors: Spiros Borotis (MENON) Contributors: Thomas Fischer, Stefania Aceto, Nikitas Kastis (MENON), Lucia Pannese (IMA), Munir Abbasi, George Xydopoulos and Masoud Fakhimi (University of Surrey)

2 Task/WP related to the Deliverable: WP7 / T7.1 Type 1 : R Reviewer(s): George Xydopoulos (University of Surrey) Approved by: (University of Surrey) Dr Munir Abbasi Contractual Date of Delivery to the EC: June 30, 2014 Actual Date of Delivery to the EC: 28 th October 2014 Actual Date of the delivery of the revised version to the EC: 7 May 2015 Abstract: This deliverable presents the Validation Plan for the CRe-AM project, detailing the topics of validation, activities and indicators. The results will be used by the project partners so as to ensure the validity of the project main outputs, and results in respect to the interest and priorities of all stakeholders. Keywords: Validation plan Project Co-ordinator Organisation name: University of Surrey Name of representative : Prof. Lampros Stergioulas Address : Surrey Business School University of Surrey, Guildford, UK, Phone number : + 44 (0) l.stergioulas@surrey.ac.uk Website : 1 Deliverable Type: P (Prototype), R (Report), O (Other) D7.1 January 2015 CRe-AM FP7 Project Grant agreement n Page 2 of 58

3 Copyright Copyright The CRe-AM Consortium D7.1 January 2015 CRe-AM FP7 Project Grant agreement n Page 3 of 58

4 The CRe-AM project consortium is composed of: SURREY University of Surrey United Kingdom MENON Reseau Menon E.E.I.G. Belgium BCC Black Cube Collective United Kingdom NCSR National Center For Scientific Research "Demokritos" Greece LMU London Metropolitan University United Kingdom IMT Institut Mines-Telecom France FLUX Fluxguide Ausstellungssysteme Og Austria Sigma Sigma Orionis France Lattanzio Lattanzio Learning SPA Italy HOPE Liverpool Hope University United Kingdom Uliv The University Of Liverpool United Kingdom IMA imaginary Italy ELIG Linkspace Management Services Gesellschaft MBH Austria Disclaimer All intellectual property rights are owned by the CRe-AM consortium members and are protected by the applicable laws. Except where otherwise specified, all document contents are: CRe-AM project - All rights reserved. Reproduction is not authorised without prior written agreement. All CRe-AM consortium members have agreed to full publication of this document. The commercial use of any information contained in this document may require a license from the owner of that information. All CRe-AM consortium members are also committed to publish accurate and up to date information and take the greatest care to do so. However, the CRe-AM consortium members cannot accept liability for any inaccuracies or omissions nor do they accept liability for any direct, indirect, special, consequential or other losses or damages of any kind arising out of the use of this information. D7.1 January 2015 CRe-AM FP7 Project Grant agreement n Page 4 of 58

5 Revision Control Version Author Date Status 0.1 Munir Abbasi and Masoud May 28, 2014 Initial Draft Fakhimi 0.2 Spiros Borotis June 11, 2014 Second Draft, Methodology 0.3 Spiros Borotis August 8, 2014 Final Draft 0.4 Georgios Xydopoulos October 9, 2014 Review 0.5 Spiros Borotis October 17, 2014 Addressing review comments, alignment with D7.2, final version 1.0 Lampros Stergioulas October 23, Submission to the EC Spiros Borotis February 21, 2015 Final draft encompassing reviewers comments and recommendations 2.0 Lampros Stergioulas May 2015 Editing and submitted to EC D7.1 January 2015 CRe-AM FP7 Project Grant agreement n Page 5 of 58

6 D7.1 January 2015 CRe-AM FP7 Project Grant agreement n Page 6 of 58

7 Executive summary The present document is the deliverable of the CRe-AM project, funded by the European Commission s Directorate-General for Communications Networks, Content & Technology (DG CONNECT), under its 7th EU Framework Programme for Research and Technological Development (FP7). The objective of this document is to present the Validation Plan for the outcomes of the CRe-AM project, in respect of the foreseen objectives as they are detailed in the Description of Work (DoW). The validation plan presented in this document addresses the main topics of a validation activity, which include the validation objects, the methods and the tools, the stakeholders engagement as well as the time plan. Therefore, the main objective is to define the procedures and tools that will be used to validate the project outputs, and secondary to gain feedback so as to improve the current approaches. The validation results will be presented under D7.4 Final Impact Evaluation Report, as a final validation report is not foreseen in the DoW. The validation approach initiates with the methodology that will guide the work towards ensuring the appropriateness and the relevance of the results created by or during the CRe-AM project with the interests and needs of the (project) stakeholders. In this regard, the CRe-AM partners developed a framework, named VALTEC that follows a multi-dimensional approach; VALTEC identifies a series of validation objects: Sector, and (potentially) cross-sectoral roadmaps: these key project outputs will be developed and validated accordingly in two phases. The initial roadmaps will be validated in terms of their appropriateness and their ability to initiate the discourse between the stakeholders communities, whereas the final roadmaps will be validated in relation to their appropriateness and relevance to the interests and needs of the stakeholders as well as their reflection with the current and emerging practices, business and technology drivers, etc. Scenarios for TEC (Technology Enhanced Creativity) futures: the feasibility (i.e. plausible vs. desired) of TEC scenarios will be assessed, in conjunction with the innovation characteristics (sustaining vs. disruptive) they encompass. D7.1 January 2015 CRe-AM FP7 Project Grant agreement n Page 7 of 58

8 Sector-specific trends, strong / weak signals and emerging developments: this validation object shall be used to assess the forthcoming changes that will affect the roadmaps developed. Sector-specific gap analysis and recommendations: scenarios and signal will feed into the gap analysis, leading to a SWOT analysis. The final results which are sector specific road map recommendations will be validated through a maturity-level tool. The roadmapping platform: this is the main tool that will facilitate and support the engagement and discourse of the stakeholders. The validation exercise will be conducted with the use of appropriate tools, including mainly questionnaires, interviews with key stakeholders and focus groups. A few examples of expert profiling also, as well as the procedure for their selection is proposed. A series of criteria and metrics are presented accordingly in chapter 3, allowing the CRe-AM consortium to optimize the effort. Validation efforts will be conducted after the outputs have been matured enough and finalizedduring final phase of the project. D7.1 January 2015 CRe-AM FP7 Project Grant agreement n Page 8 of 58

9 This project has received funding from the European Union s Seventh Framework Programme for research, technological development and demonstration under grant agreement n

10 Table of Contents 1 Introduction Validation Methodology Roadmaps Scenarios for technology futures Sector-specific strong and weak signals and emerging developments Sector-specific gap analysis The roadmapping platform From theory to practice Validation methods Questionnaires Interviews Focus groups Validation tools Validation of CRe-AM Roadmaps Scenarios for technology futures Sector-specific strong and weak signals and emerging developments Sector-specific gap analysis The roadmapping platform Validation experts Timeline of Validation Appendix Indicative validation experts Questionnaire on validation of technologies, scenarios, strong/weak signals, gap analysis and roadmapping Technology Readiness Levels (TRL) D7.1 October 2014 CRe-AM FP7 Project Grant agreement n Page 10 of 58

11 1 Introduction The scope of a validation plan is to provide the steps that are required so as to produce valid results. In this regard, the validation plan for the CRe-AM project focuses on the definition of the procedures that will be used primarily to validate the key project outputs, and secondarily to empower the key tools used to produce them. In detail, validation in the context of the CRe-AM project refers to the appropriateness and the verification of the project outcomes for the project s target stakeholder communities. Thus this effort focuses mainly on the validation of the sector-specific roadmaps. In parallel, validation has serious interrelationships with other project activities that are strongly related to the creation and refinement of the roadmaps throughout the project lifecycle. Thus, the validation effort should focus also on the outputs of WP1, WP2, WP3, WP4 and WP5. Special attention should be undertaken so as not to overlap with the quality assurance objectives and procedures that are detailed in D8.2 Project Quality and Evaluation Plan. The document is organised as follows. Chapter two presents the methodology for validation Chapter three presents the validation objects and metrics Chapter four presents two plans for timelines D7.1 October 2014 CRe-AM FP7 Project Grant agreement n Page 11 of 58

12 2 Validation Methodology The challenging nature of the project necessitates validation encompassing various dimensions for the activities conducted under the CRe-AM roadmapping efforts; this work will be conducted in parallel with the Quality Assurance, which focuses entirely on the internal stakeholders and the processes employed, i.e. the project development (i.e. project management and decision making, deliverables, project monitoring and corrective actions, as well as the internal communication mechanisms). The validation aims mainly to the external stakeholders of the project in respect to the project outputs. Thus, the aforementioned discussion sets the baseline and guides the CRe-AM consortium to define the necessary validation activities that will ensure the appropriateness and the relevance of the results created for the external stakeholders. Validation is a critical part of the development of the work under the CRe-AM project. The challenging nature of bringing into the arena the totally different by nature communities of technology developers and creative industry stakeholders (in general, including also the policy makers) so as to enhance and sustain the cooperation between them, with the aim to (co-)create better and more appropriate ICT tools for the job of the latter, necessitates the establishment of a coherent plan. Thus, the validation plan should be based on the following principles: The validation efforts should be performed on the basis of a coherent methodological framework and according to the established CRe-AM procedures. The (validation) methodology established should focus on the whole process of developing the CRe-AM roadmaps so as to ensure the priorities of adaptability, agility and dynamic roadmapping. Thus, particular emphasis should be placed firstly on the project outputs that lead to the development of the roadmaps, and secondly on the procedure (value chain) that guides the roadmap development. WHY: Validation is essential so as to ensure the appropriateness and relevance of the sector-specific roadmaps produced by the CRe-AM project with the needs and priorities of the engaged communities. For this purpose, the methodological framework should take into account a variety of different variables, like D7.1 October 2014 CRe-AM FP7 Project Grant agreement n Page 12 of 58

13 The outcomes of the CRe-AM project, e.g. the sector specific roadmaps. The usage context of these technologies. The target group / stakeholders. The gradual development of roadmaps, i.e. the initial fast-track roadmaps per sector and the final roadmaps. The project outputs that feed the roadmapping development, i.e. the roadmapping platform (WP2), the scenarios for the technology futures in the creative industry (WP3), the foresight activities outputs (WP4), the Gap Analysis (WP5), and finally Cre-AM s Creative Sector Roadmaps (WP5). The procedures employed, i.e. the methodologies used in the separate project activities 2. The requirement of a validation framework for the CRe-AM roadmaps and future scenarios motivated the researchers to delve into the literature so as to identify the relevant validation dimensions; unfortunately, most studies focused on individuals, including but not limited to the creative personality (Dollinger et al., 2004) 3, on the working climate (Isaksen et al., ; Mathisen et al., ), on activities about work and leisure (Richards et al., 1988) 6, on teachers creativity (Soh, 2011) 7, etc. Thus, the research did not indicate any generic framework or instruments that could support the validation of co-created roadmaps for innovative ICT tools developed for the needs of the creative industries. In the CRe-AM DoW, it was foreseen that the validation framework for the CRe-AM project should be based on the VALNET framework, which was originally developed for the validation of innovations related to ICT learning ICT-enabled initiatives for School of Tomorrow. The framework recognises five dimensions of innovation: Technological innovation 2 The reader should keep in mind that CRe-AM is not a research project therefore it is based mostly on already developed tools (methodologies) custom applied to its particular needs 3 Dollinger, S.J., Urban, K.K. and James, T.A. (2004). Creativity and openness: Further validation of two creative product measures. Creativity Research Journal, Vol. 16, Issue 1, pp Isaksen, S.G., Lauer, K.J., Ekvall, G. and Britz, A. (2001). Perceptions of the best and worst climates for creativity: Preliminary validation evidence for the situational outlook questionnaire. Creativity Research Journal, Vol. 13, Issue 2, pp Mathisen, G.E., Einarsen, S., Jørstad, K. and Brønnick, K.S. (2004). Climate for work group creativity and innovation: Norwegian validation of the team climate inventory. Scandinavian Journal of Psychology, Vol. 45, Issue 5, pp Richards, R., Kinney, D.K., Benet, M. and Merzel, A.P. (1988). Assessing everyday creativity: Characteristics of the Lifetime Creativity Scales and validation with three large samples. Journal of Personality and Social Psychology, Vol. 54, No. 3, pp Soh, K.-C. (2011). Indexing creativity fostering teacher behavior: a preliminary validation study. The Journal of Creative Behavior, Vol. 34, Issue 2, pp D7.1 October 2014 CRe-AM FP7 Project Grant agreement n Page 13 of 58

14 Organisational / Institutional innovation Pedagogical innovation Economic innovation Linguistic and cultural innovation and five validation objects: Projects Working processes within the project Product / resources Services Results Despite the fact that most of these innovation dimensions and overarching validation objects are adequately generic so as to be implemented in other environments / cases, they can be misleading as they focus by nature on the issue of learning; thus, the original plan of adapting the VALNET framework for the purposes of the CRe-AM project, after various unsuccessful efforts, was neglected. Therefore, for the purposes of the CRe-AM project, the Validation for Technology Enhanced Creativity (VALTEC) is developed and presented accordingly. The validation framework for the CRe-AM project should help the CRe-AM researchers and project partners to answer the following questions: What has to be validated and on the basis of what criteria (validation objects and criteria). Who will provide the required information and data (validation sources). How the validation activities will be conducted (validation method and tools). When the validation activities will be conducted (validation timeframe). As it was stated earlier, the validation efforts should focus on the main project outputs employed, so as to ensure the appropriateness and the relevance of the results with the foreseen project objectives as well as the interests of the different stakeholders. Thus, the validation objects (related to the respective tasks) that were identified and presented in detail in the following sections are: D7.1 October 2014 CRe-AM FP7 Project Grant agreement n Page 14 of 58

15 Validation Objects related to Project Outputs Validation Objects related to procedures employed Sector-specific roadmaps (T5.2) Roadmapping methodology (T1.1) (potential) Cross-sector roadmaps (T5.2) Scenarios for technology futures (T3.2) Sector-specific strong and weak signals and emerging developments (T ) Sector-specific Gap analysis (T5.1) Roadmapping platform (T2.2) Table 1 - Validation objects Thus, the VALTEC framework for validation of the CRe-AM project outputs and processes is formulated as follows: Figure 1 The VALTEC framework STAKEHOLDERS 2.1 Roadmaps The main project output comprises the technology roadmaps for each of the examined sectors of the creative industry. The roadmaps are developed in two phases; the initial (fast-track Year 1) roadmaps will become available in Month 12 and will be used as a basis for stepwise refinement, leading to the final D7.1 October 2014 CRe-AM FP7 Project Grant agreement n Page 15 of 58

16 roadmaps in Month 23. The roadmaps will be mostly sector-specific, whereas there are some degrees of freedom to produce additional (cross-sector) roadmaps. As stated in D1.1, the initial fast-track roadmaps for the creative subsectors will be used to start a discourse with the involved communities on a larger scale, proceeding to a refinement, and finally to the final roadmaps. A particular interest for the project is the potential of building synergies among the different key stakeholders. Thus, the initial fast-track sector specific roadmaps should be validated in terms of their capability to capture the interest of the different stakeholders and the appropriateness and relevance to the current and emerging practices, business and technology drivers. On the other hand, the final roadmaps should be validated in terms of the objective of developing, enhancing and mainstreaming new ICT technologies and tools that address the needs of the different sectors of the creative industries as well as the needs of the different key stakeholders. The validation results produced based on the validation object of the initial sector specific roadmaps will be used by the project to identify the strengths, the weaknesses, the opportunities and the threats of these initial roadmaps so as to enable a coherent and detailed effort in the continuous improvement towards their final versions. The validation results produced the overarching objective of cross-sector roadmaps will be used both to enhance the results of the final versions of the roadmaps and to disseminate the project achievement to all interested and related stakeholders. As stated earlier, the development of the roadmaps will be based on the desired future scenarios (developed in WP3), the strong /weak signals per sector identified in WP4, and the gap analysis conducted under WP5. In the following sections, the respective validation objects are analysed accordingly. D7.1 October 2014 CRe-AM FP7 Project Grant agreement n Page 16 of 58

17 2.2 Scenarios for technology futures The desired future scenarios will emerge through the work conducted under WP3: Collective Roadmapping: Visions and scenarios for the future of ICT in the creative industries. This work leads to a list of initial scenarios that will be either plausible or desired futures 8 and will be based on a coherent methodology developed under WP3 supported by a bunch of datasets mainly originated from the series of events (provided by the events participants) as well as other data sources (e.g. interviews with representatives from the key stakeholder groups, online survey, etc.). The scenarios will first be presented to the public in Month 12 through the online platform (contributing to the development of the initial fasttrack roadmaps), and a refined and enhanced version of them (revised scenarios) will be delivered in Month 20, leading to the final roadmaps. First and foremost, the scenarios (ideally) should be validated in terms of their feasibility (plausible vs. desired) (as they were characterised by the stakeholders, whereas the discrepancy is a part of the work of Gap Analysis T5.1) so as to infuse the time dimension into the roadmaps and enable the ICT developers and the investors to assess the risk of their efforts. Additionally, this dimension should be used by the research community in order to concentrate their efforts. Second, the scenarios should be validated in terms of the kind of (technology) innovation they require that should be fold in the continuum between sustaining and disruptive innovation (Christensen, 1997) 9. As Christensen claims, sustaining technologies focus mostly to the improvement of the performance of established products. Disruptive technologies (hereinafter mentioned mostly in the view of Christensen of the disruptive innovations so as to include also non-technology factors like new business models) although they temporarily may lead to worse product performance tend to create totally new markets or addressing the needs of new customers, disrupting in parallel existing practices or even interests. For example, the validation of the scenarios in respect to technologies identified could be conducted in a way similar to the following matrix: 8 We do not validate the feasibility of the scenarios but whether the stakeholders characterized them as plausible or desired. 9 Christensen, C.M. (1997). The innovator s dilemma. When new technologies cause great firms to fail. Harvard Business Review Press. Boston: MA. D7.1 October 2014 CRe-AM FP7 Project Grant agreement n Page 17 of 58

18 Desired scenario Plausible scenario Sustaining technologies Disruptive innovation Table 2 - The validation matrix of future scenarios This matrix should be used to validate both the initial and the revised scenarios. Concerning the initial scenarios, the results will be used by the consortium to assess whether all dimensions are covered (if not why? Is it due to lack of data or due to the needs of the sector?), and where to focus on the refinement and enhancement of the revised scenarios. They should also be used by the WP4 so as to identify appropriately the strong (related mostly to sustaining technologies) and the weak signals (related to disruptive innovations), as well as the WP5 so as facilitate the gap analysis and assessing the risk of the proposed roadmaps. The results of the validation of the revised scenarios will be used similarly by WP4 and WP5, as described above. 2.3 Sector-specific strong and weak signals and emerging developments The sector-specific strong and weak signals that will be collected under WP4: Foresight activities: trends and weak signal analysis and context scenarios as well as the results of the continuous monitoring for emerging developments that relate to the creative industries should also be validated so as to help the roadmapping process assess the risk and the applicability of the proposed roadmaps, as well as to take into account the forthcoming changes that may affect the organisation or the operation of the creative industries. Strong and weak signals will be collected in a push approach based on the data collected under the WP3 events as well as through the respective interviews and the online instruments (survey(-s), online dicsussions, etc.) conducted by the project and the partners. In a pull approach, they will emerge through the state-of-the-art analysis and other activities conducted / tools used and supported by the CRe-AM D7.1 October 2014 CRe-AM FP7 Project Grant agreement n Page 18 of 58

19 technology observatory (e.g. Google trends, for more please visit the D1.2 Conceptual Framework for Technology Observatory ). The main questions pertinent to this stage are: (a) Whether the strong signals are really strong, at least in the particular sector of interest, (b) What makes the weak signals as real signals for the sector of interest (e.g. if a technology is mentioned once, why is it a signal? Who is the person that mentioned it?) (c) Do the identified signals refer to the actual needs of the creative sector of interest or they just reflect the beliefs / developments of the ICT market? The results of this analysis will be used to validate the outcomes of the strong/weak signals collection analysis by the WP4. Last, WP4 intends to monitor continuously and analyze the social, economic, political and environmental developments in relation to emerging collaborative developments in the creative industries (Task 4.2). The results of this effort will be mapped in the report on emerging new technologies and their impact on the creative industry (D4.2). The objectives of this study are mainly to identify the (outer) drivers that emerge and will have an impact on creative industries, whether this refers to tools or emerging technologies. The validation efforts for this analysis should mainly focus on the process that is conducted (analyzed in a different validation object) and to the results as such, i.e. whether they identified what they should have, as well as if they will actually have an impact to emerging ICT developments for TEC. 2.4 Sector-specific gap analysis The results of the sector-specific gap analysis will be produced based on the work of WP5 Gap analysis and research directions. Gap analysis will take into account (a) the future scenarios and visions produced under WP3 (desired futures) and (b) the state-of-the-art analysis on strong / weak signals and emerging developments (WP4), and will try to identify how close (or not) the expected and desired (technologyenabled) scenarios proposed by the creative industries are to the emerging developments already under way in the ICT sphere, or indirectly produced upon various other motivations. And this distance estimation of the identified scenarios should be done in a maturity-level continuum (at least on the basis of technology maturity levels) so as to facilitate all interested parties to undertake particular action, taking into account also the results of a SWOT analysis of each separate (technology-enabled) scenario in direct D7.1 October 2014 CRe-AM FP7 Project Grant agreement n Page 19 of 58

20 respect to the identified needs of the creative stakeholders for each sector of interest. Certainly, the gap analysis cannot be only a SWOT analysis of technologies, as the proposed scenarios (should) include also new working practices, business models, reflecting in parallel emerging trends originating from the social, economic, political, legal and environmental upcoming developments. As described briefly above, the validation of the gap analysis will be conducted through a maturity-level tool. The literature points to various tools for this purpose, typically under the umbrella of Technology Readiness (e.g. Parasuraman, ). Typically these tools include a number of questions (TRI has 36 questions) with the results analyzed through advanced techniques, requiring a mass of data. Given the complexity of the project (too many sectors with too many technologies for each one, as derived from WP3 and WP4), this approach seems too demanding for the validation purposes. Thus, a simplified approach where the project (internally) will be employed to validate the results of the gap analysis in a maturity level scale should be employed. For the purposes of the CRe-AM project, a good idea is to be based on the wellestablished scale of Technology Readiness Levels (TRL), used by the US Department of Defense and already adopted by the European Commission for the purposes of the Horizon 2020 Programme 11. Last, in case there are available cross-sector scenarios, the gap analysis should be conducted also for them, taking into account the data obtained for the sectors of interest. 2.5 The roadmapping platform The roadmapping platform, available at is the main tool that will facilitate the continuous engagement of the key stakeholders in the roadmapping effort, as well as the stepwise refinement and continuous improvement of the sector-specific (and possibly the cross-sector) roadmaps. As mentioned in the Description of Work This tool is established with the aim to support the roadmapping community engagement and management as well as to empower and encourage the discourse among ICT and creative stakeholders. Thus, the platform should be validated in terms of the aforementioned objectives. 10 Parasuraman, A. (2000). Technology research index (TRI). A multiple-item scale to measure readiness to embrace new technologies. Journal of Service Research, Vol. 2, No. 4, pp D7.1 October 2014 CRe-AM FP7 Project Grant agreement n Page 20 of 58

21 3 From theory to practice This chapter presents in detail the scheduling and tools for the application of the validation methodology presented in the previous chapter. In detail, a set of validation methods, tools accompanied with criteria and metrics, as well as expert profiles examples are presented accordingly. 3.1 Validation methods The validation will be conducted through the employment of various methods, based on the topic of validation. These include questionnaires, interviews and focus groups (workshops) Questionnaires Structured questionnaires with open and close-ended questions will allow respondents to express their personal beliefs and provide rich insights for the topics of interest. These tools available offline and / or online based on the locus of engagement (during workshops or in the online roadmapping space of the CRe-AM project) will allow for the presentation of individual beliefs in scales of the form strongly agree strongly disagree, providing also the option of no answer to the respondent. Each questionnaire should be pilot-tested before going public. The questionnaires should be developed by the responsible partners for each task, or the sector facilitators respectively. Indicative indicators have been presented earlier and indicative questions are presented later in this chapter Interviews The interviews for validation purposes should be done with particular people that have specific knowledge on the topic of interest; either it is related to the project outputs, or to methodological issues. The interviews should be based on structured or semi-structured questionnaires, developed by the partners responsible partners for each different working topic, or the sector facilitators respectively. Indicative indicators have been presented earlier and indicative questions are presented later in this chapter Focus groups Focus groups will be used by the validation team to assess the validity both of the project outputs and the methodologies employed, in offline face-to-face sessions. This technique is essential so as to establish D7.1 October 2014 CRe-AM FP7 Project Grant agreement n Page 21 of 58

22 interaction between the engaged stakeholders. Please see D1.2 Conceptual framework for technology observatory for a detailed description of the method. 3.2 Validation tools In this section, the validation objects analysed in the previous section are assigned with the appropriate criteria, metrics, target group and timeline. It is worth mentioning here that the indicators and metrics are indicative and will be refined by the project prior to application Validation of CRe-AM Roadmaps As described in the methodology, the roadmaps will be validated in two phases. First, the initial fast-track sector-specific roadmaps will be validated so as to enable further refinement and potential revisions. The criteria and the respective metrics, as well as the target group and the timeline are presented in the following tables. Validation Object Initial sector-specific roadmap Criteria Degree of engagement of the communities Appropriateness of the initial sector-specific roadmaps to the current and emerging practice Relevance of the initial sector-specific roadmaps to the current and emerging business and technology drivers SWOT analysis Table 3 - Object: Initial sector-specific roadmap Criterion: Degree of engagement of stakeholders Metric The CRe-AM roadmapping activities enhanced and facilitated the collaborative work between communities of creators and communities of technology/tools providers without conflicting with their respective rights Answered by Table 4 - Criterion: Degree of engagement of stakeholders in initial sector-specific roadmapping effort D7.1 October 2014 CRe-AM FP7 Project Grant agreement n Page 22 of 58

23 Criterion: Appropriateness of the initial sector-specific roadmaps to the current and emerging practice Metric Answered by The proposed roadmap reflects the needs and the priorities of your sector for the current and emerging practice I like the proposed roadmap Table 5 - Criterion: Appropriateness of the initial sector-specific roadmaps to the current and emerging practice Criterion: Relevance of the initial sector-specific roadmaps to the current and emerging business and technology drivers Metric The proposed roadmap is relevant to the current and emerging business practices and technology drivers I find the proposed roadmap useful for current and future working practice Answered by Table 6 - Criterion: Relevance of the initial sector-specific roadmaps to the current and emerging business and technology drivers Criterion: SWOT analysis Metric In your opinion, what are the strengths of the proposed roadmap? In your opinion, what are the weaknesses of the proposed roadmap? In your opinion, what are the opportunities for your sector if the proposed roadmap comes true? In your opinion, what are the threats for your work if the proposed roadmap comes true? Table 7 - Criterion: SWOT analysis of the initial fast-track sector specific roadmaps Answered by Next, concerning the final roadmaps, the respective criteria, and mechanics are as follows Validation Object Final sector-specific roadmap Criteria Degree of engagement of the communities Degree of relevance of the roadmap to the needs of the (different) stakeholders SWOT analysis Table 8 - Object: Final sector-specific roadmap Criterion: Degree of engagement of stakeholders Metric Answered by The CRe-AM roadmapping activities enhanced and facilitated the collaborative work between communities of creators and communities of technology/tools providers without conflicting with their respective rights Table 9 - Criterion: Degree of engagement of stakeholders to the final sector-specific roadmapping effort D7.1 October 2014 CRe-AM FP7 Project Grant agreement n Page 23 of 58

24 Criterion: Relevance of the roadmap to the needs of the stakeholder Metric The proposed roadmap is relevant to the my current and emerging needs and priorities Table 10 - Criterion: Relevance of the final sector-specific roadmap to the needs of the stakeholder Answered by Criterion: SWOT analysis Metric In your opinion, what are the strengths of the proposed roadmap? In your opinion, what are the weaknesses of the proposed roadmap? In your opinion, what are the opportunities for your sector if the proposed roadmap comes true? In your opinion, what are the threats for your work if the proposed roadmap comes true? Table 11 - Criterion: SWOT analysis of the final sector-specific roadmaps Answered by Last, the overarching validation object of cross-sector roadmap is presented in case such roadmaps emerge through the process. Validation Object Cross sector roadmap (if they appear) Criteria Degree of engagement of communities Appropriateness of the cross-sector roadmap to the current and emerging practice of the sectors engaged Relevance of the roadmap to the needs of the stakeholder SWOT analysis Table 12 - Object: Cross-sector roadmap Criterion: Degree of engagement of communities Metric The CRe-AM roadmapping activities enhanced and facilitated the collaborative work between communities of creators and communities of technology/tools providers without conflicting with their respective rights Table 13 - Criterion: Degree of engagement of stakeholders to the cross-sector roadmapping effort Answered by D7.1 October 2014 CRe-AM FP7 Project Grant agreement n Page 24 of 58

25 Criterion: Appropriateness of the cross sector roadmap to the current and emerging practice of the sectors engaged Metric The proposed roadmap reflects the needs and the priorities of the relevant sectors for the current and emerging practice I like the proposed roadmap Table 14 - Criterion: Appropriateness of the cross sector roadmap to the current and emerging practice Answered by Criterion: Relevance of the roadmap to the needs of the stakeholder Metric The proposed roadmap is relevant to my current and emerging needs and priorities Table 15 - Criterion: Relevance of the cross sector roadmap to the needs of the stakeholder Answered by Criterion: SWOT analysis Metric In your opinion, what are the strengths of the proposed roadmap? In your opinion, what are the weaknesses of the proposed roadmap? In your opinion, what are the opportunities for your sector if the proposed roadmap comes true? In your opinion, what are the threats for your work if the proposed roadmap comes true? Table 16 - Criterion: SWOT analysis of the cross sector roadmap Answered by Scenarios for technology futures The scenarios emerging from the WP3 should be validated according to their feasibility as it was expressed by the stakeholders, so as allow the ICT sector to focus appropriately its efforts. Validation Object Scenario Criteria Feasibility and applicability of the future scenario SWOT analysis Nature of proposed technology / innovation Table 17 - Object: Scenario for technology future Criterion: Feasibility of the scenario Metric Position of the proposed scenario in the continuum between plausible and unachievable Table 18 - Criterion: Feasibility of the proposed scenario Answered by D7.1 October 2014 CRe-AM FP7 Project Grant agreement n Page 25 of 58

26 Criterion: SWOT analysis of the scenario Metric In your opinion, what are the strengths of the proposed scenario? In your opinion, what are the weaknesses of the proposed scenario? In your opinion, what are the opportunities for your sector if the proposed scenario comes true? In your opinion, what are the threats for your work if the proposed scenario comes true? Table 19 - Criterion: SWOT analysis of the scenario Answered by Criterion: Nature of proposed technology / innovation Metric Position of the proposed technology / innovation in the continuum between sustainable and disruptive Do you agree with the proposed scenario? Can you propose another one that better suit your priorities? Table 20 - Criterion: Nature of proposed technology / innovation Answered by Overall, the proposed scenarios should be placed in the following matrix Desired scenario Plausible scenario Scenario 2 Scenario 1 Scenario 4 Scenario 5 Scenario 3 Sustaining technologies Disruptive innovation Table 21 - Example of placing the proposed scenarios in the validation matrix of scenarios for technology futures Sector-specific strong and weak signals and emerging developments The results of the WP4 efforts on strong/weak signals per sector could be validated in terms of the significance of the outcomes (i.e. whether strong signals are really strong, the weak signals are signals overall, and whether they refer to the actual needs of the sector of interest). The emerging (social, economic, political, environmental) developments will be validated in terms of the hypothesis that they consider outer drivers that will have an impact on the future of the European creative industry. D7.1 October 2014 CRe-AM FP7 Project Grant agreement n Page 26 of 58

27 Validation Object Sector-specific strong and weak signals Criteria Validity of the strong / weak signals identified Degree of reflection of actual needs of the sector of interest Table 22 - Object: Sector-specific strong and weak signals Criterion: Validity of the strong / weak signals identified Metric Number of times (and overall percentage) a signal mentioned Table 23 - Criterion: Soundness / popularity of the strong signals Answered by Partners Criterion: Reflection of actual needs of the sector of interest Metric The proposed signal reflects the actual needs of the sector of interest Table 24 - Criterion: Validity of the weak signals identified Answered by Validation Object Outer drivers that will have an impact on emerging ICT developments for TEC Criteria Coverage of the outer drivers for emerging ICT developments for TEC Table 25 - Object: Outer drivers that will have an impact on emerging ICT developments for TEC Criterion: Coverage of the outer drivers for emerging ICT developments for TEC Metric The study indicated that the following issues ( ) emerging from the social, economic, political and environmental development will affect the development of new technologies for your sector. How these emerging issues could affect your work? Table 26 - Criterion: Outer drivers for emerging ICT developments for TEC Answered by D7.1 October 2014 CRe-AM FP7 Project Grant agreement n Page 27 of 58

28 3.2.4 Sector-specific gap analysis The sector-specific gap analysis will build on the findings of WP3 and WP4 so as to identify the gaps between the future scenarios and the current and emerging trends. As described in the particular section of the methodology, the results of it will be validated by the project partners through a maturity-level continuum (at least for the technologies proposed). Validation Object Sector-specific gap analysis Criteria Validation of the gap analysis results based on the Technology Readiness Levels (TRL) scale Table 27 - Object: Sector-specific gap analysis In this case, the TRI scale presented in the Appendix could be used by the consortium to identify the position of each proposed technology identified in the TRI scale. An additional textbox could be used for comments. Criterion: Technology Readiness Index 12 Metric The study identified a list of technologies that appear to be as useful and desired for your sector the following years. For each one of these, please rate it in terms of its maturity. Answered by Partners Table 28 - Criterion: TRI for gap analysis In case the consortium decides to create cross-sector roadmaps apart from the sector specific then a cross-sector gap analysis should be also conducted. In this case, the same criterion may be used The roadmapping platform The roadmapping platform developed under the CRe-AM project will facilitate the validation of the efforts of the CRe-AM project, as online engagement can reveal a lot about the views/attitudes of the participating stakeholders. 12 This validation activity could be conducted only for the final version of the roadmaps D7.1 October 2014 CRe-AM FP7 Project Grant agreement n Page 28 of 58

29 Validation Object Roadmapping platform Criteria Degree of engagement of stakeholders to the roadmapping process Degree of satisfaction and discourse among ICT and creative stakeholders Table 29 - Object: Roadmapping platform Criterion: Engagement of stakeholders to the roadmapping process Metric Number of stakeholders engaged overall and per sector (in terms of the expected ones by the signed LOIs) # of posts/blogs/resources/comments each user of the platform have contributed in different working groups. Average time that the different users spent on the online platform Rate of creative ideas generated from the participation in various working groups on CRe- AM platform. Daily/monthly number of unique visits to the platform Platform availability and down-time, weekly or monthly number of running time errors, service breakdowns or unreachable resources Table 30 - Criterion: Engagement of stakeholders to the roadmapping process Answered by Partners Partners Partners Partners Partners Partners Criterion: Satisfaction and discourse among ICT and creative stakeholders Metric Level of awareness of the users about CRe-AM platform features and tools. The level of satisfaction among users on efficiency of the CRe-AM platform to address the needs of different sectors of creative industries. Level of efficiency and accessibility of the platform for different user across EU Table 31 - Criterion: Satisfaction and discourse among ICT and creative stakeholders Answered by The validation activities will be coordinated and reported from the respective leader with the extensive support of the partners allocated the work of T7.1: Development of Validation Activities. The validation results will be presented under D7.4 Final Impact Evaluation Report, as it is not foreseen a final validation report in the DoW. 3.3 Validation experts A key cornerstone of the validation activities is the experts that will be engaged to this endeavour; these people will originate from all the different types of stakeholders, enabling a coherent representation, so as D7.1 October 2014 CRe-AM FP7 Project Grant agreement n Page 29 of 58

30 to maximize the validity of the results eventually produced. The CRe-AM consortium will identify a first bunch of validation experts before conducting the validation activities, expanding them accordingly based on the needs of every particular activity. Experts profiles Experts for the purpose of CRE-AM validation are defined as people with extensive knowledge or ability based on research outputs, experience, or occupation in the particular creative sector of interest. Thus, the validation experts that would participate to the validation efforts of CRE-AM should have deep knowledge on the state-of-the-art of their sector, especially in relation to technology (e.g. being a high level staff of a creative company). All types of stakeholders should be represented to the validation team of experts. This effort will be conducted based on the following process: Step 1: Identify the initial bunch of validation experts. These people will originate from the list of the already engaged stakeholders who present deep and proven knowledge of creative technologies on each separate sector or in a cross-sectional level, and have raised already their interest on CRe-AM efforts and results. The identification will be facilitated by the sector(-s) facilitators and co-facilitators. Step 2: Expand the list of validation experts. The team of validation experts will be expanded accordingly taking into account the list of stakeholders that have originally signed a LoI, probably having registered themselves to the roadmapping platform (not required) but never engaged in practice to the CRe-AM activities. The experts engaged in Step 2 could be more oriented to sectors. Step 3: Snowball sampling. The validation team will be expanded gradually and horizontally (sector-specific) employing the snowball technique originating from the statistical sampling. Already engaged stakeholders as well as the members of the validation experts team will be requested to propose people with extensive knowledge or abilities based on research outputs, experience or occupation in the particular creative sector of interest. Step 4: Exploiting the network of each separate partner, all partners will focus on their academic and industry links, to provide a balanced approach. D7.1 October 2014 CRe-AM FP7 Project Grant agreement n Page 30 of 58

31 In the appendix, a list of experts identified through the CRe-AM engaged stakeholders is presented accordingly. D7.1 October 2014 CRe-AM FP7 Project Grant agreement n Page 31 of 58

32 This project has received funding from the European Union s Seventh Framework Programme for research, technological development and demonstration under grant agreement n Timeline of Validation A coherent validation effort in such a complex project like CRe-AM necessitates the continuous engagement of stakeholders, in each phase, so as to obtain timely results and undertake corrective measures. Month Tool (-s) WP related Month 13 - One questionnaire validating technologies, initial scenarios, strong/weak signals, gap analysis and roadmaps, filled in by 10 key experts from each sector - Roadmapping platform questionnaire (in collaboration with T7.2) 3,4,5 2 1 Month 24 - One questionnaire validating technologies, final scenarios, emerging ICT developments for TEC, gap analysis and roadmaps, filled in by 10 key experts from each sector - Roadmapping platform questionnaire (in collaboration with T7.2) 3, 4, 5 3,4,5 2 Table 32 Validation Plan Timeline

33 5 Appendix 5.1 Indicative validation experts In this section we provide indicative short profiles of people that could be engaged as experts in the validation efforts, identified from the list of CRe-AM stakeholders. The reader should keep in mind that these profiles are only indicative. Cross-sectional indicative validation experts: Dave Colpin is currently Microsoft's Chief Envisioning Officer. Dave is passionate about turning the base metal of technology into valuable assets that effect the way that we live, learn, work and play and in so doing, move the focus from the technology itself to the actual outcome. Dave has the goal of making technology less "visible" and more valuable in our daily lives. Dave regularly questions how technology is shaping how we think and how we work. In his recent book, The Rise of The Humans: How to Outsmart the Digital Deluge, Dave argues that right now it feels like the machines are taking over but if we stop thinking about the digital deluge as a problem and instead see it as an incredible opportunity we will be able to redress the balance. Technology offers our society so much but it is up to us, the humans, to rise to that potential. Dave could be included in the first bunch of validation experts, encompassing also a crosssectoral role. More information on Dave can be found on the following: Peter Friess is the IoT Project Officer (DG Connect - European Commission, a senior official of the EC taking care for more than five years of the research and innovation policy for the Internet of Things. In his function he has shaped the on-going European research and innovation program on the Internet of Things and accompanied the European Commission's direct investment for around 70 Mill. Euro in this field. As part of the Commission Internet of Things European Action Plan from 2009, he also oversees the D7.1 October 2014 CRe-AM FP7 Project Grant agreement n Page 33 of 58