EUROPEAN COMMISSION

Brussels, 6.9.2018

SWD(2018) 398 final

COMMISSION STAFF WORKING DOCUMENT

on the evaluation of Europeana

Accompanying the document

Report from the Commission to the European Parliament and the Council

on the evaluation of Europeana and the way forward

{COM(2018) 612 final}

Table of contents

1.Introduction

2.Background to Europeana

3.State of Play

4.Method

5.Analysis and answers to the evaluation questions

6.Conclusions13

Annex 1: Procedural information15

Annex 2: Stakeholder Consultation (synopsis report)17

Annex 3: Evaluation Questions23

Glossary

1.Introduction

Purpose and scope

On 31 May 2016, the Education, Youth, Culture and Sports (EYCS) Council adopted conclusions on the role of Europeana for the digital access, visibility and use of European cultural heritage 1 . The Council Conclusions confirmed Member States’ continued support for Europeana, recognised Europeana’s relevance both from a cultural as well as a digital innovation perspective, identified challenges to be addressed, and invited Member States, the Commission and the Europeana operator to undertake action.

In these Council Conclusions, the Commission was invited:

‘to present to the Council an independent evaluation of Europeana and give clear orientations for the mid- and long-term development of Europeana by assessing alternatives at the EU level for the future scope, sustainable funding and governance of Europeana, including a possibility to transform or integrate Europeana into a European legal entity, whilst taking account of the dual nature of Europeana as both a cultural and digital innovation project;’

In response to the Council’s request, the Commission carried out an evaluation of Europeana based on the five mandatory criteria of the Commission’s Better Regulation Guidelines 2 . Europeana was assessed in terms of its (i) relevance, (ii) effectiveness, (iii) efficiency, (iv) coherence, and (v) EU added value. The scope of the evaluation covered both the underlying concept and value of Europeana as a European cultural and digital innovation project, as well as an in-depth review of the adequacy and viability of the services provided by Europeana as a Digital Service Infrastructure (DSI) under the Connecting Europe Facility (CEF) 3 .

The evaluation results provided input to an analysis of future options and to the formulation of orientations for its mid- and long-term development, as requested by the Council.

The outcomes of the evaluation will be transmitted to the Council and Parliament and will further inform future activities of the Commission.

2.Background to Europeana

Europeana was launched in 2008 as ‘The European Digital Library’ to make Europe's cultural heritage. Its creation was initiated by a letter from six Heads of State and Government to the Commission in 2005. Financial and political support has continued ever since 4 for coordinating and integrating EU Member States’ efforts to digitise and make cultural material accessible online, including standardisation, interoperability and cooperation among cultural heritage institutions across Europe. Europeana contributes to the implementation of a number of policy areas. These include the Digital Single Market, the Commission Recommendation of 27 October 2011 on the digitisation and online accessibility of cultural material and digital preservation (2011/711/EU) 5 and related Council Conclusions (EYC Council, May 2012) 6 , calling on Member States to get more material online and ensure the long-term preservation of digital material.

Today, Europeana is Europe’s digital platform for cultural heritage, funded as a well-established Digital Service Infrastructure (DSI) under the Connecting Europe Facility (CEF), to:

-bring Europe’s digitised cultural heritage material online and to promote its cross-border visibility and use across Europe;

-offer solutions for the trans-European interoperability and accessibility of digital resources of European heritage;

-further improve the conditions for re-use and facilitate take-up of digital cultural content and metadata in other sectors, e.g. research, education, tourism or the creative industries; and

-provide a multilingual, user-friendly access point to Europe’s rich and diverse heritage.

Europeana offers services creating value for different user groups:

-data partners (contributing cultural institutions) and cultural heritage professionals;

-end-users (anyone with an interest in culture);

-re-users (education, research and creative sector professionals).

Europeana promotes standardisation, best practice and capacity building among cultural heritage institutions and professionals in the area of cultural and digital innovation.

Europeana contributes to preserving and fostering European culture in the digital age and involves all EU Member States. It supports the objectives of the European Year of Cultural Heritage 2018 to encourage the sharing and appreciation of Europe’s cultural heritage, to raise awareness of common history and values, and to foster a sense of belonging to a common European space. Europeana is the only digital cultural heritage partner of the European Year of Cultural Heritage 2018.

3.State of Play

Europeana currently provides access to over 51 million items from the collections of over 3 700 libraries, archives, museums, galleries and audio-visual collections across Europe, through its main portal ‘Europeana Collections’ ( europeana.eu ) and through the platform’s application programming interfaces (APIs). The material that is accessible through Europeana includes books, manuscripts, audio and video recordings, national archives, documents illustrating Europe's history, manuscripts of major writers and composers, as well as digital copies of artworks.

Following the Council Conclusions of 2016, Europeana refined its Strategy 2020 and focused on three priorities:

-Making it easy and rewarding for cultural heritage institutions to share high quality content.

-Scaling with partners to reach target markets and audiences.

-Engaging people on Europeana’s websites and via participatory campaigns.

A consortium led by the Europeana Foundation is operating the current Europeana DSI core service, i.e. the central platform and services. Europeana Foundation cooperates with (i) a number of domain and thematic aggregators that support the aggregation of material from cultural heritage institutions, (ii) partners supporting distribution and re-use of Europeana data, and (iii) expert partners supporting different technical aspects. A number of national aggregators support Europeana in improving the aggregation infrastructure and data quality. The Europeana Foundation has led a series of consortia operating the Europeana initiative since its launch in 2008. A number of user-oriented projects, supporting thematic collections, tools and re-use, have been funded as generic services since 2016.

From 2008 to August 2017, Europeana was funded through grants. Its scope and main objectives were set in the annual work programmes of the funding instruments. Throughout this period, the implementation strategy of Europeana was largely set out by Europeana Foundation, as leader of the winning consortia.

Since grants require co-financing, the winning consortia had to identify additional sources of revenue to cover all the costs necessary to run Europeana. Voluntary Member States' contributions have always been a challenge as Member States considered that their contribution was through digitisation of content and contribution of that content to Europeana.

The Commission has made provisions for stable continuous support to the Europeana platform. Following the Council Conclusions of 2016, the Commission changed, as of September 2017, the funding model for Europeana’s core platform and services to one of procurement while continuing to fund generic services projects and activities through grants.

The Europeana Foundation Governing Board of stakeholders and experts from the cultural heritage community meets five times per year to set the implementation strategy and review plans on goals and objectives.

The Europeana Network Association is a community of cultural heritage, creative, and technology professionals, who exchange and promote best practices and cross-border cooperation. Through its Members Council, it provides input on Europeana’s strategy and activities. Its Management Board (six members elected by the Members Council) represents the Europeana Network Association on the Governing Board of the Europeana Foundation.

Member States influence the direction of Europeana through the CEF Telecom Committee, the Commission’s Expert Group on ‘Digital Cultural Heritage and Europeana’, and participating on the Europeana Foundation Governing Board.

4.Method

The evaluation of Europeana was conducted through the following process:

·An independent comprehensive study was carried out from July 2017 to May 2018 for gathering and analysing data and for assessing the evaluation questions. The tasks of the study included refining the evaluation questions and methodology, data gathering (including desk research, case studies, interviews, benchmarking, technical usability evaluation and analysis of the public online consultation), and assessing the evaluation questions.

·A panel of five independent experts, from different fields of relevant expertise, were appointed by the Commission 7 . The experts validated the evaluation questions and provided feedback, comments and a final assessment.    

·The Commission also carried out a 12-week public online consultation, open between 17 October 2017 and 14 January 2018, in all EU official languages. A targeted questionnaire, with more technical questions, addressed to professionals and organisations that had experience with Europeana’s infrastructure and professional network was included in the public consultation. The public consultation results have provided significant input to the evaluation.

·The entire process was followed by an Interservice Steering Group with representatives from different Commission DG to ensure the quality and impartiality of the process.

5.Analysis and answers to the evaluation questions

The main findings of the evaluation, based on the five evaluation criteria, and corresponding evaluation questions, can be summarised by the following.

5.1.Relevance

The evaluation has highlighted that, with over 51 million items, Europeana provides access to one of the largest digital collections of cultural material in the world. Europeana further stands out by offering material in a large number of languages (37 languages), as well as in the variety of its content (text, image, 3D, audio-visual, sound) from a broad network of data providers from different domains (e.g. archives, libraries, museums). The benchmark study also shows that Europeana targets a wider scope of audiences (cultural heritage institutions and professionals; research; education; creative industries; culture professionals; individuals) than other cultural heritage platforms analysed. However, this makes it more difficult for Europeana to satisfy every user group’s needs.

Europeana’s relevance to EU policies and priorities for the online accessibility and dissemination of European cultural heritage is rated high. The concept and core values of Europeana fully support EU policy 8 by bringing the common cultural heritage to the fore, enabling European cooperation, and expressing an inclusive narrative about the diversity and interconnectivity of culture in Europe. Furthermore, Europeana supports the EU Digital Single Market policies related to online accessibility and digital preservation of cultural heritage by improving the availability and re-use of digital cultural heritage across sectors and across national borders.

According to the findings of the case studies and the public consultation, Europeana’s services are relevant to cultural heritage institutions for the online accessibility and dissemination of European cultural heritage. Specifically for small cultural heritage institutions, the possibility for them to share data in Europeana is perceived as beneficial because it increases the visibility and recognition of their collections.

Europeana’s relevance as a facilitator of professional exchange of cultural and technological expertise is recognised in the findings of the case studies and the public consultation analysis. Europeana’s past and current activities to find common solutions and to set up frameworks (such as the Europeana Publishing Framework and the international standardisation work under the RightsStatements.org initiative) are seen as very important for the digital cultural heritage sector. Cultural institutions benefit from best practices as far as common standards are concerned and from joining and learning from a pan-European network.

While Europeana has exceeded its objectives in terms of quantity of items the relevance and quality of its content and metadata is an issue. Although at least two thirds of the public consultation’s respondents stated they were satisfied with various aspects of Europeana’s content, most responses to the open question elaborating on this answer provided constructive criticism. These mainly pointed towards the need for more high quality material and for material from currently underrepresented European countries, certain institutions or of specific themes. They also stated that the quality of the content was unsatisfactory, particularly the resolution and colours of images. Rights statements were also considered to be restrictive in some cases and therefore to preclude re-use.

In terms of metadata, the lack of translation, its varying quality (poor or incorrect metadata), and inconsistencies (often due to different metadata formats used by different institutions) have a negative impact on the service’s overall popularity and affect the findability of an object when searched for on Europeana. A number of negative responses in the public consultation concerned the lack of relevance of search results. High-level quality data is a key parameter for Europeana, and cultural heritage institutions play a central role in this regard, being ultimately responsible for the quality of the provided content and metadata. The recent update of Europeana’s strategy, focusing on strengthening the quality of data and making the existing (and incoming) content on Europeana more interesting and valuable to end-users and re-users, is highly relevant.

5.2.Effectiveness

Europeana reaches a significant number of user profiles, whose interests in the platform are quite diverse. Public consultation results indicate that most respondents are from the education, culture and research sectors.

The diversity of users and their interests in Europeana provide a challenge in maintaining high quality services for all targeted groups.

The evaluation study shows that the current performance indicators of Europeana are not sufficient for monitoring progress. Moreover, the statistics and analytics covering previous years are not readily accessible nor presented in a systematic manner. Statistics must be published on a regular basis in order to provide the most up-to-date data to data partners and other stakeholders. It is recommended that further data gathering on reach and scope is considered, firstly, to identify where further investigation can be focused and, secondly, as a way of segmenting Europeana’s performance with the different audiences.

In terms of technical infrastructure, Europeana is on a par with the other platforms in the benchmark, offering the interfaces and functionality expected from a hub for the exchange of cultural information. According to the benchmark analysis, Europeana is on a par with the other examined platforms in aspects such as interoperability with data providers and re-users; aggregation of metadata and content, procedures for metadata import and mapping, data management and multilingual indexing. However, there is room for significantly improving the aggregation infrastructure and workflow, and the functionality of the Europeana portal.

One of Europeana’s main achievements was to be able to gather thousands of individuals and organisations (in particular cultural heritage institutions) from very different backgrounds and enable them cooperate in a dynamic and expanding Europeana Network to share metadata about their content in a single place. However, the findings highlight the need to (i) make the aggregation process easier for data providers, (ii) reduce the time from data submission to publication, as well as (iii) provide better tools and technical support for the data ingestion process. Case study interviewees stated that these actions are very important for engaging and motivating the cultural heritage institutions, which in turn are crucial for increasing the overall data quality.

Those who answered the question on the extent to which they were satisfied with finding what they searched for were asked to elaborate on their answer. The negative responses pointed mainly to the lack of features on the website and the lack of relevance of the search results. Specifically, respondents pointed to a lack of sophisticated filtering as a method of refining a search and a need for more advanced searching possibilities for the portal and the platform’s APIs. Respondents also reported search results to be inconsistent, containing broken links, lacking thumbnails, and containing duplicates or redundant hits.

Another issue is the portal’s level of multilingualism: Only 46 % of respondents to the public consultation indicated satisfaction with multilingualism of metadata. The majority (59 %) of the respondents indicated that they use English for their searches. Respondents pointed specifically to the lack of translation of metadata and to inconsistencies in search results when searching in different languages.

The well-curated collections on the portal have significantly higher satisfaction and engagement rates. Additionally, user engagement campaigns like Europeana 1914-1918 (collection of family stories and memorabilia related to WW1 provided by the public) proved successful and innovative in reaching and engaging a wider audience in the locations where they took place.

API services were seen as very interesting by the developers that took part in the usability evaluation. Nevertheless the search engine did not always provide relevant results or the results contained broken or blank links. The API documentation was sometimes found to be incomplete or inaccurate.

One of Europeana’s greatest achievements lies in providing standards and frameworks regarding the publication and sharing of data that have been taken up in the cultural heritage sector across the EU and internationally, for example by the Digital Public Library of America (DPLA). This is, in particular, the case of the Europeana Data Model (EDM), the Europeana Publishing Framework (EPF) and the RightsStatements.org international initiative. The case study on the Europeana Publishing Framework shows that its use has brought progress on the overall quality of data, although there is still a lot of scope for improvement since the biggest part of the material (around 64 %) is currently in Tier 1 (lowest tier out of four).

On communication activities, 21 % of the respondents wanted to see more efforts made to raise awareness about Europeana. Respondents mainly emphasised that awareness-raising activities should, on the one hand target the general public, for instance, through greater presence on social media and improved search-engine optimisation. On the other hand, many respondents also highlighted students and researchers as key groups to be targeted for awareness-raising activities, as these are main re-users of Europeana’s content.

In terms of adapting to a fast changing technological and user-driven environment, Europeana’s technical infrastructure is progressively improving, through the continuous re-design and improvement of its platform and portal. From a technical point of view, the data model, the publishing framework and the interoperability procedures offer a basis for further developments. The main challenge is the speed and effectiveness with which the Europeana organisation and governance structure will be able to react to new requirements. Technical review reports covering the period under evaluation show examples of where plans and investments started but did not conclude, or only did so in a minor way. Europeana’s ability to adapt to change is also influenced by the fact that it relies on data provided by participating content providers. The challenges these providers have in transforming their collections in a rapidly changing technological environment are inherited by Europeana. The case study on small and medium sized institutions shows that Europeana adds value in upgrading knowledge, skills and tools at cultural heritage institutions level.

5.3.Efficiency

Europeana’s infrastructure faces a number of challenges and constraints regarding efficiency.

A major challenge is the current aggregation infrastructure. It is based on domain and national aggregation, which has the benefit of dealing only with a couple of hundred direct partners to bring in data from 3 700 institutions. The drawbacks are the complex structure and content submission process and the amount of time needed from data submission to publication, often leading to a lack of motivation for data providers.

Of the respondents that provided data to Europeana, only around half indicated satisfaction with the various aspects of Europeana’s aggregation structure. This question was elaborated upon in an open answer, with 76 % providing constructive criticism and suggestions for improvement. 11 % of the responses stated they were dissatisfied with the inability to update content and/or metadata after it had been provided to Europeana, and 9 % stated they were dissatisfied with the communication with Europeana regarding support and feedback on the data provision process. This was particularly the case for some cultural heritage organisations that had contributed to Europeana through EU-funded projects that have now ended, often leading to a ‘publish and forget’ practice.

Case study analysis indicates that domain aggregators (including best practice networks such as AthenaPlus, EU Screen or Photoconsortium) and national aggregators help cultural institutions to share their data and guarantee a correct provision of data to Europeana. This assistance is considered as particularly beneficial for smaller cultural institutions. The case study also highlighted that the national aggregators are better equipped to survive and link to Europeana even though not every country has one.

Despite the above-mentioned limitations, Europeana notably managed to build a wide ecosystem or network of aggregators and cultural heritage institutions that play a key role in providing input at both national and European level. Europeana’s governance involves a large part of the initiative’s stakeholders and is well established to handle changing challenges while remaining relatively efficient. Some limitations relate to the need for Europeana to be more inclusive and to ensure a greater participation of aggregators and cultural heritage institutions, as well as to clarify the role and responsibilities of all players involved in the organisation.

The funding model for Europeana’s core service platform based on grants did not provide a sufficiently stable basis for funding Europeana, since it required co-funding from other sources. Despite the efforts of the Commission, the Member States and the Europeana Foundation, the grant model proved to be unsuitable for sustaining the core service. In the 2016 Council Conclusions, Member States reached an agreement to continue their contributions to Europeana until the Commission progressively converts its support scheme from a grant model to a procurement model that could cover all core service costs. This shift took place in September 2017.

Since it started in 2007, consecutive EU funding programmes (eContentplus (2005-2008), Competitiveness and Innovation Programme (2007-2013), CEF (2014-2020)) have contributed to the development and deployment of the Europeana initiative. The funding has supported both the central coordination and the European integration of national efforts to make digital cultural material accessible through Europeana, including standardisation, interoperability and targeted digitisation.

Funding of approximately EUR 10 million a year under the current CEF programme focuses on maintaining and further developing Europeana’s core platform service, while also supporting, since 2016, user-oriented projects as generic services. The EU funding has been entirely used to support the development and deployment of the Europeana infrastructure. Europeana is not an initiative associated with regulatory costs, savings, burden reduction in Member States or similar, but serves as a pan-European set of digital and innovation services for cultural heritage that would otherwise not exist. As such, the assessment of Europeana is based on the benefits and impact that the initiative has achieved.

Quantitative data on the socio-economic benefits and impact generated by Europeana is not available. However, the Europeana initiative has had a high leverage factor: whereas the Commission supported directly around 900 organisations to make their collections accessible through Europeana, and to carry out targeted digitisation, the Europeana initiative has raised awareness among the cultural heritage sector on the importance of having a digital presence, has advanced cultural heritage institutions’ digital expertise and has mobilised cultural institutions to make their material accessible online. This has led more than 3 700 institutions to contribute to Europeana (leverage factor of 4:1) and over 51 million European cultural heritage items can be accessed digitally through Europeana.

5.4.Coherence

The evaluation confirms the coherence of Europeana as regards wider EU policy, in line with the Gothenburg communication of November 2017 on Strengthening European Identity through Education and Culture 10 . With regard to strengthening the sense of European identity and awareness of cultural heritage, the Gothenburg communication designates the 2018 European Year of Cultural Heritage as the occasion to show how digital tools can broaden access to cultural material and opportunities, and to shed light on initiatives such as Europeana, which fosters access to cultural heritage material held by libraries, archives and museums through digital means.

Europeana is overall coherent with the national strategies for the digitisation and online accessibility of cultural heritage material of the Member States. Europeana, to a certain extent, addresses challenges that the EU Member States face in digitising cultural heritage and transforming access to culture. It complements national initiatives with similar objectives and is the only pan-European platform of its kind.

Europeana’s work on, for instance on standards, tools and open access, is seen as useful across all sectors influenced by digital transformations, even if not all stakeholders agree to the specific solutions promoted.

Europeana maintains overall internal coherence, its activities and services being consistent with each other, even though evaluation findings state that further effort should go into improving certain aspects, such as internal communications, having a sufficient number of technical staff, focusing on enabling core technology and narrowing the scope of activities.

5.5.EU Added Value

According to the evaluation study, Europeana has played a significant role in the digital development of the cultural heritage sector in Europe. Case study analysis indicates that Europeana’s efforts have been key to strengthening cooperation and standardisation activities across borders in line with the overarching goal of increasing the use of standards in digitising and sharing digitised cultural material throughout Europe. The Europeana Network of aggregators, cultural heritage institutions and professionals is seen as an important component of cross-border and pan-European collaboration, leading to interoperability and standardisation in the digital cultural heritage sector.

Overall, projects developed under the Europeana initiative enabled new developments to emerge at both national and European level.

The main topic raised by the positive comments was that Europeana has underpinned a sense of a shared history and identity among European citizens (for example, ‘Unifying European history, reminding us that we have a long and rich shared past not without problems [ …] Europeana’s importance is obviously more than a platform with pretty images’).

6.Conclusions

The evaluation has highlighted an overall appreciation of the European added value of Europeana by cultural heritage institutions across the EU. Europeana is a unique initiative and the only pan-European platform of its kind. It is the only aggregator in the world that brings together the largest digital collection from across 28 Member States. It has also helped inspire a sense of shared history and identity among European citizens through its unique collections, such as the popular 1914-1918 collection recalling the First World War. At the same time, it has brought together European cultural institutions, enabling them to collaborate and share their material with European citizens. Not only has this led to sharing of best practices on common standards, but also it has helped nurture a European network of data partners, aggregators and professionals in various fields who have stimulated the capacity building and exchange of expertise. Some of the major achievements are the progress towards a de facto standardisation in the cultural heritage sector though the Europeana Data Model, the Europeana Publishing Framework and the International Rights Statements.

Europeana also complements national strategies of Member States for the digitisation and online accessibility of cultural heritage, and reflects EU policy, such as the overall objectives of the Gothenburg communication of November 2017 on Strengthening European Identity through Education and Culture.

However, the evaluation results also highlight that several transformations are needed in a number of areas. For example, there are significant challenges for Europeana stemming from its focus on a large number of diverse activities appealing to a wide range of user groups. Narrowing down the focus of efforts and concentrating resources to reach maximum quality and impact of the offered services would be necessary.

Better quality of services remains a persistent demand from many respondents of the open public consultation, be they cultural heritage institutions or individual users of Europeana. Quality related issues are interlinked between quality of content and metadata, the findability of material, and multilingualism functions. The findability of content would be improved if more metadata was translated or was available in more than one language.

Strengthening the technical platform core is a condition sine qua non for Europeana to remain a reliable cultural content and service provider. The evaluation further highlights the need to strengthen the aggregation infrastructure for Europeana to be able to service and support the cultural heritage institutions in their actions towards capacity building and digital transformation.

Annex 1: Procedural information

1.Lead DG, Decide Planning/CWP references

This Evaluation Staff Working Document was prepared by Directorate G ‘Data’ / Unit G.2 ‘Interactive Technologies, Digital for Culture and Education’ of Directorate-General ‘Communications Networks, Content and Technology’ (DG CNECT).

The Decide planning reference of the initiative ‘Evaluation of Europeana’ is PLAN/2016/55.

2.Organisation and timing

·A panel of five independent experts from different fields of relevant expertise were hired to assist the Commission in evaluating Europeana and assessing possible options for its future development.

Names of the experts (alphabetically):

-Prof Dieter Fellner, Director, Fraunhofer Institute for Computer Graphics Research IGD, Germany;

-Ms Laure Kaltenbach, Creative Entrepreneur, co-founder of CreativeTech Agency and ex-Managing Director, Forum d’Avignon, France;

-Dr Jarred McGinnis, Independent Consultant in Semantics, United Kingdom;

-Dr Piotr Rypson, Deputy Director of the National Museum in Warsaw, Poland;

-Ms Gianna Tsakou, Project Manager/Senior Analyst, SingularLogic S.A., Greece.

·The Commission procured an independent study 11 which was carried out by a consortium led by the consultancy firm CARSA 12  between July 2017 and May 2018. The tasks of the study included:

orefining the evaluation questions and methodology;

ogathering and analysing evidence data; and

oassessing the evaluation questions and drafting the report.

·An Interservice Steering Group (ISG) with representatives from DG Communications Networks, Content and Technology, DG Education, Youth, Sport and Culture, DG Research and Innovation, DG Informatics, and the Secretariat-General has followed the entire process. The ISG approved the evaluation roadmap and stakeholder consultation strategy, the terms of reference and the deliverables of the external study, as well as the online questionnaire of the public consultation.

3.Evidence, sources and quality

The Commission, supported by the external contractor, gathered qualitative and quantitative evidence from the following sources.

·A public online consultation on Europeana ran between 17 October 2017 and 14 January 2018, in all EU official languages. The consultation generated 1 221 responses, of which 883 were from individuals and 338 were on behalf of an organisation. Among the individuals, 66 % indicated that their main interest in Europeana was personal use and 61 % professional use.

·A targeted questionnaire, with more technical questions, addressed to professionals and organisations that had experience with Europeana’s infrastructure and professional network was included in the public consultation.

·Literature review

·Four case studies, including two interviews per case study:

oThe value of the Europeana Network;

oEuropeana Publishing Framework — activities and impact;

oRe-use of Europeana in education; and

oBenefits for cultural institutions that share data in Europeana.

·Benchmarking between Europeana and seven other organisations that have developed digital cultural heritage projects:

oin EU Member States: the British Library, Finna.fi, BnF, Rijksmuseum;

oInternational organisations: The World Digital Library, The Digital Library of America (DPLA), Google Arts & Culture.

·Usability evaluation of the platform at three levels:

otechnical evaluation;

oevaluation with user and developer groups;

oautomated and quantified evaluation of the user’s perception in terms of availability and response time.

·The national progress reports on digitisation, online accessibility and digital preservation of cultural material submitted by Member States for the period 2015-2017. A section of the reports concerns the Member States’ contributions to the Europeana initiative.

·CEF sustainability study and CEF mid-term evaluation.

The panel of experts conducted additional informal interviews with the Executive Director and the Deputy Director of the Europeana Foundation (operator), representatives of the Europeana Foundation Governing Board and representatives of the Europeana Network Association Management Board.

Annex 2: Stakeholder Consultation (synopsis report)

1.Introduction

Following the Council Conclusions of 31 May 2016 13 , the European Commission launched an independent evaluation of Europeana ( www.europeana.eu ). The scope of the evaluation covered both the underlying concept and value of Europeana as a European cultural and digital innovation project and an in-depth review of the adequacy and viability of the services provided by Europeana as a Digital Service Infrastructure under the CEF programme.

This report summarises the outcomes of the public online consultation and of the other consultations conducted by the Commission to ensure that the opinion of individuals and organisations is well reflected in the evaluation as well as in the Commission’s proposal for Europeana’s future.

2.Public consultation

The public online consultation was held, in all EU languages, between 17 October 2017 and 14 January 2018.

All organisations and members of the public with a personal or professional interest in digital cultural heritage, or in Europeana in particular, were invited to express their opinion, experience and expectations on Europeana and the various aspects of the platform.

A targeted questionnaire was included in the public online consultation. It was addressed to professionals and organisations working in cultural heritage, research, education and the creative sector that had already used Europeana as a platform for sharing or re-using cultural heritage content.

Respondents profile

Of the 1 221 14 respondents that took part in the online consultation, 883 responded as individuals (72 %) and 338 responded on behalf of an organisation (28 %).

Culture, research and education were the most represented sectors for both individuals and organisations. Museums and libraries were the most represented culture organisations, together covering more than 70 % of respondents from that field. Some 63 % of those responding on behalf of educational organisations represented higher education.

Among individuals, 66 % indicated that their main interest in Europeana was for personal use and 61 % for professional use.

Use of Europeana platform — main results

Respondents agree that the portal ‘Europeana Collections’ ( europeana.eu ) is valuable in various aspects: it advances cultural knowledge and learning (82 % of respondents); Thematic collections and exhibitions allow exploration of specialised content (85 %); Trustworthiness of content sources (82 %); Availability of material of different languages (79 %).

Two thirds of respondents indicated that Europeana was important for finding and exploring European cultural heritage.

Respondents also agree that Europeana is relevant for those interested in using digital cultural heritage material in their work, particularly in providing relevant material with suitable quality for re-use (75 % of respondents), suitable licencing for re-use (65 %) and assuring the trustworthiness of its data (75 %).

Respondents replying as individuals were asked how often they used different parts of Europeana. 35 % indicated that they use the portal ‘Europeana Collections’ on at least a monthly basis. Similarly, 20 %, 15 %, and 8 % indicated that they use the professional pages of Europeana, Europeana material for other projects, and Europeana’s APIs, respectively, on at least a monthly basis. At most 8 %, at most, indicated that they use the Europeana platform on a weekly basis (‘Every day’ or ‘A few times per week’).

Respondents replying as individuals were asked what other portals they use to access cultural heritage information. Overall, 58 % of the respondents use Wikipedia on a weekly basis (‘Every day’ or ‘A few times per week’), followed by the portals of national institutions (22 %) and the Internet Archive (21 %)

Out of 201 respondents that elaborated on the aspects they particularly like in cultural heritage portals which they could not find on Europeana (i) 19 stated that they seek higher or consistent quality of content and another 20 that they seek content pertaining to specific themes; (ii) 69 pointed to a better search functionality, especially on the relevance of the search hits; (iii) 33 pointed to the quality of the metadata, such as the level of detail of the descriptions; and (iv) 12 respondents pointed to clearer, as well as less restrictive licencing.

On Europeana’s campaigns and activities to engage citizens with culture, among the individuals that responded to the consultation, 9 % of respondents were aware and had engaged with the ‘Collection days’ campaign, while 49 % were aware of the campaign but did not engage. Apart from this campaign, the majority (65 %) of respondents were unaware of other Europeana campaigns and activities.

User experience with Europeana — main results

Questions related to the user experience with the Europeana platform were addressed to respondents replying as individuals.

The majority indicated that they seek images on Europeana (88 %), followed by texts (64 %). 59 % use English for their searches. 41 % indicated that they often find what they search for while another 44 % indicated that they sometimes find what they search for.

In relation to the user experience when searching on Europeana the following information was obtained.

·57 % of the respondents (452 out of 794) were satisfied with the ease of searching and filtering results, 48 % with the ease of navigating through multilingual material, and 53 % with the relevance and accuracy of results.

·45 % of the respondents (182 out of 404) were satisfied with the ease of using the APIs.

·69 respondents described problems when searching on Europeana. They pointed mainly to a lack of website features, such as the level of multilingualism, the lack of sophisticated filtering for refining search results, a need for more advanced searching possibilities, and a lack of relevance of the search results (inconsistencies, broken links, lack of thumbnails and duplicates or redundant hits in the search results).

In terms of satisfaction with various aspects of the content and metadata:

·the percentage of respondents that indicated satisfaction with the content on Europeana ranged between 50 % and 63 % in terms of: (i) geographical coverage, (ii) thematic coverage, (iii) quantity, (iv) technical quality, (v) uniqueness, and (vi) re-usability;

·71 respondents provided constructive criticism about Europeana’s content, pointing mainly to the content quantity, quality, and licencing;

·The percentage of respondents that indicated satisfaction with Europeana’s metadata ranged between 52 % and 62 % in terms of: (i) completeness, (ii) detail, (iii) accuracy, (iv) relevance, and (v) links to original objects. Only 46 % indicated satisfaction with the multilingualism of metadata (availability of metadata in their language);

·65 respondents provided constructive criticism regarding issues on the availability of metadata in multiple languages, metadata quality and consistency;

Data partners

310 respondents, replying on behalf of an organisation or as individuals with a professional interest in Europeana, had collaborated with Europeana as a data partner and took part in this targeted section. Most respondents believe it is important to provide content to Europeana and indicated that they were likely to continue supplying data via an aggregator (84 %) or directly (58 %) in the future. Lack of time or resources (31 %) and lack of information on how to provide data (25 %) were indicated as the largest barriers to providing content.

The highest proportion of data partners (82 %) agree that Europeana’s promotion of open culture/metadata and content has facilitated their work within their organisation, while at least two thirds of data partners indicated that Europeana generated value for their organisation via audience reach (72 %), added value to content (74 %), and facilitation of partnerships with other cultural institutions (65 %).

The percentage of data partners indicating satisfaction with Europeana’s aggregation structure ranged between 35 % and 44 % in terms of: (i) the time it takes from source to publication, (ii) the level of automation along the aggregation chain, (iii) communication, (iv) the quality assurance process, (v) support, and (vi) overall effort to provide data. Some 40 respondents elaborated on their answer in text. Negative responses mainly highlighted a lack of tools in the aggregation process, notably for content providers to easily publish and update their content, as well as a lack of fluidity of communication between Europeana and the aggregators or the providers in order to resolve issues and facilitate the publication of data. Of the positive responses, the main point highlighted was satisfaction with the technical support provided by aggregators to the content providers.

Re-use of material

89 respondents replying on behalf of an organisation and 534 replying as individuals, who indicated that they had, at least once or twice, re-used material from Europeana, took part in the targeted section on re-use. The highest proportion of respondents indicated that they re-used content from Europeana in research projects or as teaching materials.

72 respondents elaborated in text on how they have re-used the material. Four main themes emerged: in presentations (17 respondents); in school or academia, such as in student assignments (15 respondents); on the web, e.g. in blogs, social media, and the integration of the Europeana API into other portals (13 respondents); and for personal recreation (8 respondents).

In relation to awareness of Europeana’s activities on engaging re-use professionals, the most well-known activity was the Europeana hackathons. 221 respondents (52 %) indicated that they were aware of such activity, although only a small proportion (less than 7 %) indicated that they engaged in a hackathon. Most respondents claimed to be unaware of other activities.

Europeana Network Association (ENA)

222 respondents took part in the targeted section regarding engagement with the activities of the ENA, replying as individuals or on behalf of an organisation. Of these, about two thirds indicated that they participate at least occasionally in the elections for the Members’ Council, in the Annual General Meetings and in working groups or task forces.

Respondents indicated that the ENA has enabled them to contact professionals and experts they would not otherwise be able to find (71 %), that such exchanges have been beneficial to them (70 %), and that the ENA complements other digital cultural heritage networks (72 %). More than half indicated that it is easy to interact with ENA members and that they feel they had a say in how Europeana is developed.

14 out of the 60 respondents that provided comments about participating in the ENA felt that their contributions were not taken into account by Europeana decision makers.

EU added value

At least two thirds of the respondents agreed that Europeana brings value to the EU, particularly in digitally bringing together and providing access to cultural objects from across Europe (85 %), standardising the format of object descriptions (70 %), and facilitating a European network of digital heritage professionals (70 %).

30 respondents elaborated on ways that Europeana has created value on the European level. The main positive topic raised was that Europeana had underpinned a sense of a shared history and identity among European citizens.

158 respondents elaborated on national, European, or international developments that they thought would not have happened had Europeana not existed, such as:

·national or institutional level initiatives, such as the creation of online cultural heritage portals based on Europeana’s model, or initiatives that aggregate and provide content to Europeana (11 %);

·European developments: Europeana’s contribution to the digitisation and online accessibility of content (6 %) and improved cooperation within the cultural heritage sector (18 %);

·international collaborative efforts to which Europeana has contributed (2 %);

·other initiatives, which have been enabled thanks to Europeana’s efforts such as the establishment of licencing standards and liberalisation of licences (8 %).

Forward looking questions

In the last section of the consultation, 585 respondents expressed their views on the most significant issues and priorities for Europeana. The following topics emerged.

·Website functionality:

·dissatisfaction with Europeana’s current search function, for instance hits with low relevance, limited filtering of search results (14 %);

·higher level of multilingualism on the Europeana website; metadata provided not only in the native language (7 %).

·Communication activities:

·more efforts to raise awareness and visibility of Europeana to the general public, as well as to teachers, students and researchers (21 %).

·Content on Europeana:

·quantity: wish for more content from underrepresented countries, of specific themes and different formats (15 %);

·quality: more quality content, such as images in higher resolution and without watermarks (7 %).

·Licencing of the content — respondents indicated a wish for:

·Europeana to urge content providers to make their content available under less restrictive licencing, particularly for non-commercial use (8 %);

·greater clarity and accuracy in the re-use declarations associated with the content on Europeana (4 %).

·Metadata of Europeana’s content — respondents wish for:

·semantic enrichment of the metadata, better linking of the data, and stronger enforcement of metadata standards (6 %);

·complete, accurate and detailed metadata (5 %).

·Aggregation and data ingestion:

·respondents indicated a wish for more technical support to data providers (5 %).

395 respondents expressed views on what new features, services, or activities could be introduced by Europeana in the future, including:

·in-text search and advanced filtering (13 %);

·tools such as smartphone apps, download facilities, virtual exhibition creators, and games (7 %);

·tools allowing users to contribute to Europeana by correcting metadata, social tagging, or commenting (5 %);

·improved multilingualism by machine translation (4 %); and

·tools for data providers to upload content and update metadata (5 %).

3.Other stakeholder consultation actions

In addition to the online consultation, the following consultation actions were carried out:

·During a four-week period, all interested stakeholders were able to provide feedback on the published Evaluation Roadmap 15 , outlining the overall evaluation approach. The Europeana Foundation, the Europeana Network Association and the Flemish government submitted comments.

·Eight case study interviews were conducted under the evaluation study, on:

o‘The value of the Europeana Network’;

o‘Benefits for cultural institutions that share data in Europeana’;

o‘Europeana Publishing Framework (EPF) — activities and impact’; and

o‘Re-use of Europeana in education’.

The input received through these interviews confirmed the perceived value of Europeana and its network of aggregators and cultural institutions for the interoperability and online accessibility of cultural material as well as for the development of standards and common solutions. Visibility, and recognition and being part of a larger ecosystem were mentioned as the main motivations for cultural institutions to share data in Europeana.

Interviews pointed out that it is important to get the data providers more involved, by developing the infrastructure to facilitate data provision and reduce the complex and time-consuming workflows for publication. Continuous support provided by Europeana’s national and/or domain aggregators and expert hubs was perceived as very important in order to reach and support data providers, particularly small institutions.

Interviews also highlighted an increased awareness of the problem of multilingualism. Some cultural institutions do not have the capacity to share metadata in a language other than their native language, which negatively affects the quality and findability of the material.

Efforts to explore re-use of Europeana collections in education — either integrated into normal teaching in the classroom, or through integration in national educational resources portals to contribute the European perspective — showed promising results and good potential to have a larger impact. However, the problems of finding suitable content in the appropriate language were reported as hindering wider integration and the re-use of Europeana collections in education.

·As an extension to the case study on benefits for cultural institutions that share data in Europeana, a short survey was completed by nine small cultural institutions in Belgium, Germany and Italy. Two national institutions in Luxembourg were also invited to complete this short survey, since no organisation from Luxembourg participated in the online consultation. Input received through these questionnaires coincided with the input collected through the interviews for this case study.

Annex 3: Evaluation Questions

The evaluation roadmap set out the following questions to be addressed by the evaluation. The evaluation questions were validated during the evaluation process.

Relevance

·To what extent is the concept of Europeana relevant to the needs of the EU? Are Europeana’s outputs and results relevant to EU priorities, in particular the flowering of cultures of the Member States (Article 167 TFEU) and the Digital Single Market?

·Are Europeana’s services relevant to the needs of its user groups: data partners, end users, re-users?

·How relevant is the content and metadata?

·How relevant is Europeana as a facilitator of professional exchange of cultural and technological expertise in the digital heritage community?

Effectiveness

·How successful have the offered services been in meeting requirements of the different audiences? How effective is Europeana in reaching end-users and re-users?

·What has been the progress towards achieving an impact, based, where applicable, on statistics and performance indicators? Are the current performance indicators of Europeana sufficient for monitoring progress?

·Has the underlying technical infrastructure, data model and functionality of Europeana been adequate for achieving its objectives?

·Does Europeana’s governance ensure the most inclusive and effective operation?

·To what extent have the presentation of results and outputs of Europeana to stakeholders and the public been effective?

·How does Europeana meet the requirements of a fast changing technological and user environment?

Efficiency

·How efficient is the current aggregation infrastructure?

·How efficient is the current governance structure?

·To what extent is the current support scheme efficient to respond to the needs of Europeana and its stakeholders?

·How do the costs involved compare to the achieved results and impact?

Coherence

·How coherent are Europeana’s activities and services internally?

·To what extent is Europeana coherent with national strategies in Member States and with other national or international initiatives with similar objectives (e.g. national aggregators such as Gallica, Hispana or German Digital Library; Research infrastructures such as DARIAH or E-RIHS; or DPLA)

·To what extent is Europeana coherent with wider EU policy, including with the European Agenda for Culture in 2007 whereby cultural heritage has been a priority under successive Council Work Plans for Culture?

EU added value

·What is the EU added value of Europeana compared to what could be achieved by the private sector or by Member States at national, regional and/or institutional level?

·Have there been national, European or international developments that would not have happened without Europeana?

(1)   The role of Europeana for the digital access, visibility and use of European cultural heritage — Council conclusions (31 May 2016)

(2) European Commission Better regulation: guidelines and toolbox

(3)   Evaluation Roadmap

(4)  The timeline of the digitisation and online accessibility of cultural heritage outlines all relevant initiatives and policy documents since 2005.

(5)   Commission Recommendation of 27 October 2011 on the digitisation and online accessibility of cultural material and digital preservation (OJ L 283. 29.10.2011, p. 39)

(6)   Council Conclusions of 10 May 2012 on the digitisation and online accessibility of cultural material and digital preservation (OJ C 169, 15.6.2012, p. 5).

(7) The panel of experts were also tasked with analysing scenarios and options for the future development of Europeana, based on the evaluation outcomes. See Annex I.

(8)   Treaty on the Functioning of the European Union (TFEU), Article 167.

(9)   https://www.europeana.eu/

(10)   Strengthening European Identity through Education and Culture, COM(2017) 673 final, 14.11.2017

(11) EU Bookshop: Evaluation of Europeana and orientations for its future development, following adoption of Council Conclusions by EYCS Council on 31/05/2016 (SMART no 2016/0100)

(12) Consortium composed of CARSA (lead partner), Ramboll Management Consulting, Agilis and SQS (subcontractor).

(13)   Council Conclusions of 31/5/2016 on the role of Europeana for the digital access, visibility and use of European cultural heritage (OJ C 212, 14.6.2016, p. 9).

(14) Overall, the online consultation received 1 226 responses, of which 5 duplicates were deleted during the clean-up process. The resulting dataset consists of 1 221 responses.

(15)   http://ec.europa.eu/smart-regulation/roadmaps/docs/plan_2016_55_europeana_evaluation_en.pdf