This deliverable describes the outcomes of the external evaluation of GoNano project, and, based on this experience, suggests a semi-qualitative methodology for the evaluation of co-creation processes and activities.
The evaluation was done with reference to the expected impacts of GoNano, based on specific indicators addressing the efficacy and efficiency of the process, and the efficacy of the project. These indicators have been measured using a mixed methodological approach, by means of both primary (interviews) and secondary (e.g., project deliverables) sources. Results have been analyzed and included in this report.
Overall, the quantified outcomes of the evaluation show that the GoNano project succeeded in terms of both efficacy and efficiency, and thus impacts of the action. The most relevant outcomes related to the three expected impacts of GoNano are:
- Impact 1 (product level). Realize R&I products aligned with social needs and values. The project was successful in delivering a wide range of products ideas and suggestions, aligned with the views of the stakeholders engaged in the process. However, the level of maturity, structure and definition of these products varies a lot, and only few of them might find a practice use in the future. The effective engagement of industry and policy makers has been a bottleneck of the process. The project was effective in taking into account gender and diversity, which were mainstreamed all along the process and the project, either in the setting up of the co-creation process and in its implementation.
- Impact 2 (policy level). Support more responsive approaches to strategy building and policy making in R&I. This action has been affected by the difficulties in engaging with policy makers and industrial stakeholders during the project, also due to changes in the planning of events due to the covid19 situation. However, it has also been realized that this impact was quite ambitious given the methodology and activities planned in the project. In order to achieve this goal, it is important that policy and decision-makers go beyond responsiveness to societal needs and values, and societal engagement, and put forward initiatives to motivate and incentivize research, business and industry to change their way of working.
- Impact 3 (stakeholder level). Facilitate inclusive approaches for R&I, to increase confidence
on R&I. The project was very successful in terms of favoring the mutual learning and active engagement of different actors. There were good and successful foundations in the project for opinions and voices to be heard: results were very good in terms of interaction between different stakeholders and in increasing trust among them. Providing evidence of the potential (economic) benefits for product development of co-creation is still an open issue, that might go beyond the goals of the project
Some reflections on how to perform co-creation emerged from the external evaluation:
- Fostering mutual learning: co-creation should be implemented both as an “external process”,engaging a diversity of stakeholders (e.g., citizens, professional stakeholders), and as “internalprocess”, ensuring interactions and mutual learning within the teams/partners coordinating the co-creation process
- Defining expectations: co-creation can have different goals, ranging from the democratization of science to user-led innovation. It is key to understand and address this dichotomy since the very beginning of the process
- Ensuring engagement and motivation of industry and policy actors: these stakeholders are essential, as they are the ones that could put in practices results of the co-creation process, and thus enhance responsiveness of innovation. They need to be strongly involved, motivated and incentivized, and be proactive in the co-creation process
- Continuous engagement: stakeholders need to be engaged throughout the co-creation
process - Review and adapt methodologies: the approach to co-creation should be adapted to the specific context, and thus methodology design and implementation go hand-in-hand: methodology developers directly involved “on field” in the engagement activities, for testingthe efficacy of the process and fine-tune it, whenever necessary.
The methodology adopted for the evaluation of GoNano, proved to be applicable along all the three phases of the co-creation process, namely:
- Identification and structuring of the problem,
- Dealing with the problem,
- Implementation.
The proposed approach went beyond the assessment of the deliverables and KPIs defined in the project itself, and while building upon the praxis available in literature, it was integrated with the following additional aspects:
- Adopting a co-creation approach for the evaluation method itself, by involving the project’s partners in the design of the methodology and indicators
- Defining specific indicators for capturing the quality of the outcome and of the impacts.
It was built upon the concept of collecting information and interacting only with the mediators of the co-creation process (i.e., the project’s partners), and not with the stakeholders participating in the process. This has been possible given that each of the specific co-creation actions performed by individual partners in GoNano already foresaw extensive collection of feedbacks from involved stakeholders.
In addition, it was pointed out the importance of involving the external evaluator in selected cocreation events, to collect primary information on the process itself, and to triangulate the feedback and data collected by the partners.
As a next step, it would be interesting to apply the proposed approach in other co-creation actions, so to fine-tune and improve it, with the ambitious and ultimate goal to define a common approach to the evaluation of the efficacy, efficiency and impact of co-creation processes.
Read the full report here.