As explained, the process evaluation relied on several data sources. De-spite this, we draw attention to the fact that there are three main limita-tions in the empirical data that should be noted; 1) access to data thru the national contact persons; 2) Lack of written documentation from the cases; 3) English as working language of Nordic development projects.
The design of the process evaluation defines the joint meetings and input from the national cases as the main data source. Due to both the organisation of the cases and the budget, the national contact persons became the main source of information on the national context and ex-periences from the involved cases between the joint meetings. There have not been resources to visit national cases or local project, and lim-ited possibilities to provide other kinds of studies of the issues raised by the Nordic project. In addition to the joint meetings, the design declared the national contact persons as our main source of data from the involved national cases. Thru the process we have had to extend the emphasis on document studies and search for research and policy documents to obtain more information on the context of the national and local cases from each country. Many of the questions raised in the evaluation are, as pointed out in both the interim reports, difficult to answer for national contact persons or to provide information on. The projects are anchored in the education sector and collaboration with other sectors at the na-tional level has to a limited degree been established. In the first phase of the project, this was reflected in difficulties in responding to or providing information on systems and relevant services anchored in other sectors (such as health and social affairs) (Hansen et al. 2018:24). We have met this limitation by providing information thru document studies and other
Faforeport 2020:21
42
level, there is a lack of empirical data in this evaluation related to cross-sectoral collaboration at the national level. Many of the national contact persons have also found it difficult to answer the more detailed questions on experiences from the cases. An explanation for this may be that the national contact persons are not necessarily closely involved in the local cases as pointed out earlier in this chapter and thus have limited infor-mation/knowledge on the details of what is happening in the local cases and what learning points can be generated.
The other limitation is that even though the mapping and the discus-sions and reflections from the Nordic joint meetings have provided im-portant insights into how to achieve more relevant and collaborative practices, there is limited written documentation from the cases. Sys-tems, models and methods from the involved cases have rarely been pre-sented in a structured way as a basis for joint reflection with the aim of identifying common learning points. In the autumn of 2020, some of the national cases are working on documentation and dissemination of learning points, but this has been available only to a very limited degree throughout the process. As an example, we can mention the issue of ar-ranging interdisciplinary meetings. Many participants at the Nordic meetings have contributed to learning in relation to how to arrange in-terdisciplinary and cross-sectoral meetings as this is an issue in several of the included cases, but there is very little written documentation of methods and experiences so far. The same goes for empowering children, young persons and families in meetings with services and schools. This is addressed in most of the cases, but there are few documented methods and results. In Finland there are examples of schools where different ser-vices have been integrated, but there are limited documentations of mod-els and results. In Iceland, at Breidholt, they have started a documenta-tion and evaluadocumenta-tion of their specific model of integrated school support services in a local service centre and close collaboration with schools in the area. This is important as the model is implemented in all districts in Reykjavik, but as of September 2020 this evaluation has still not been fi-nalised. When it comes to concrete contributions related to models of funding and regulations that might promote cross-sectoral collabora-tion, we have not been able to achieve any empirical data from the in-volved cases in these matters.
A third limitation that should be noted is related to having English as a working language in these kind of Nordic development projects. Both in the mappings and in the joint meetings, having English as the working
language has been challenging. Participants who had to communicate in a second language may have restricted some of the sharing of experiences and details about involved cases. The participants were very open to help each other communicate, and if all participants in a group understood a Scandinavian language, they often switched to this. Of course, on many occasions the use of Scandinavian languages excludes those who do not understand any of them. This is a challenge in all Nordic collaboration.
The first mapping demonstrated that formulating the questions in Eng-lish might have made it more demanding to understand what we were asking for, and to respond, and as such we made amendments in the sec-ond mapping. The secsec-ond mapping was formulated both in English and Norwegian, with respondents allowed to respond in English or any Scan-dinavian language. This made the responses more thorough.
Reliability and validity
Even though we made some amendments in our approach during the evaluation process, we stayed true to the main objective of the evaluation and research questions. As pointed out, there are some limitations in the empirical data, but still we have several data sources as the basis of our analysis: We followed the presentations and discussions in the joint meetings, facilitated dedicated group discussions, conducted mapping of information from the national cases and carried out both group and in-dividual interviews with participants. It is challenging to discuss reliabil-ity and validreliabil-ity in qualitative studies such as this evaluation. The relia-bility relies on a clear design, the group of several researchers involved, and routine peer review of the reporting from the project.
There must always be a critical review of the validity related to whether results from studies could be transferred to other contexts. Early in the evaluation process we introduced some theories and concepts used in the analysis – and thru the process, have stressed the important of context in the interpretation of the findings. More than clear results, we present factors that thru the analysis have shown to be important to achieve a more coherent and effective follow-up of the target group and to improve cross-sectoral collaboration. Previously (Hansen et al. 2018; 19) we have discussed that the sample of national cases and the design do not make strictly comparable analysis possible. The aim of the evaluation was not to pinpoint best practice. Rather, thru analysis of the presentations made
Faforeport 2020:21
44
the project and, further, to highlight factors which, on the basis of the analyses of this material, stand out as important in achieving more col-laborative practice.
The transfer value from qualitative evaluations of this kind is always open to debate. In this evaluation we have also stressed the significance of context (taking context into consideration) and of having locally an-chored processes involving all stakeholders as important factors in de-velopment processes towards new approaches. The way one method works in a specific school in a municipality in Denmark, might not work in Norway. At the same time, the analysis and the evaluation has shown that there are certain factors across different initiatives involved in the Nordic 0–24 project that seem to be of great importance in succeeding with the implementation of new, more collaborative practice. Discussing the findings in the light of other studies makes the results more relevant beyond the specific cases involved in the Nordic 0–24 collaboration.