Table of contents

1.    INTRODUCTION    3

2.    BACKGROUND OF THE INTERVENTION    5

3.    IMPLEMENTATION / STATE OF PLAY    10

4.    METHOD    14

5.    ANALYSIS AND ANSWERS TO THE EVALUATION QUESTIONS    16

6.    CONCLUSIONS    47

7.    ANNEXES………………………………………………    …………….…………. 50

Glossary

1.Introduction

1.1. Purpose and scope

In an age of rapid technological progress, digital technologies bring unprecedented opportunities to the cultural heritage sector. They provide more effective tools to digitise cultural heritage assets, for diverse purposes such as preservation, conservation, restoration, reproduction, research, education, as well as for a broader, more democratic online access and reuse in key ecosystems such as sustainable tourism and the cultural and creative industries. They provide the public with numerous ways to access, discover, explore and enjoy cultural material while cultural heritage institutions can reach broader audiences, and engage them in innovative ways by offering immersive, creative and accessible content.

Cultural heritage is an important contributor to the European economy, fostering innovation, creativity and economic growth. For example, cultural tourism accounts for 40% of all tourism in Europe 1 , and cultural heritage is an essential part of cultural tourism. There are places, regions and even some countries where the local economy relies extensively or even completely on the revenue that cultural heritage sites and venues generate by attracting visitors to those areas.

Digitisation of cultural heritage and the reuse of such content can generate new jobs not only in the cultural heritage sector, but also in other key areas such as the creative industries (e.g. the video game industry, film industry). Such processes also provide incentives for companies to develop more efficient technologies, and new business models.

However, cultural heritage, including monuments, archaeological landscapes, buildings and artefacts, continues to face significant threats of deterioration or partial or full destruction or disappearance. Some threats are human-derived, e.g. mass tourism, terrorism and vandalism, while others originate in nature, e.g. deterioration over time, pollution, flooding, wind erosion. There have been regular, tragic reminders (e.g. fires at the Nantes Cathedral in 2020, at the Notre Dame in Paris in 2019, at the Cavallerizza Reale in Turin in 2019, the theft from the Green Vault at Dresden’s Royal Palace in 2019) of the need to leverage digital technologies to avoid the complete loss of the memory of our cultural heritage assets and to enable their reconstruction and reproduction.

The unprecedented challenges of the COVID-19 pandemic, with its significant social and economic consequences, have heightened even more the need for and the urgency of using digital technologies. Many cultural heritage institutions (‘CHIs’) had to close and thus incurred major financial loss. However, despite the loss of income, reaching 75-80% 2 for some CHIs, many of them have managed to bounce back 3 and keep or extend their audiences. There has been an increase in the digital services they offer 4 (e.g. engaging with audiences, sharing collections, offering digital tools), demonstrating once more their high value to society.

The Commission Recommendation on the digitisation and online accessibility of cultural material and digital preservation (2011/711/EU) 5 (“the Recommendation”), is the Commission’s main policy tool for digital cultural heritage. Its aim is to optimise the use of information and communication technologies to unlock the full economic and cultural potential of Europe’s cultural heritage.

The Recommendation dates back to October 2011. Some of the challenges facing the cultural heritage sector at that time, i.e. an urgent need to protect and preserve European cultural heritage at risk are still present today, but others have arisen since. Major advances in digital technologies for example have created new opportunities that did not exist when the Recommendation was adopted. Therefore, the Recommendation needs to be assessed in order to decide whether or not it still meets the needs and expectations of the cultural heritage sector, of the users, of the creative sectors, of society in general. In this context of major digital developments, a clear, evidence-based assessment of the Recommendation will help the Commission to decide on the follow-up steps, such as a revision of the Recommendation.

In line with the Better Regulation Guidelines 6 , this evaluation assesses how the Recommendation has been working and actually performing compared to initial expectations by examining its effectiveness, efficiency, relevance, coherence and EU added value.

This staff working document (‘SWD’) addresses the five areas in the Recommendation, i.e. organisation and funding of digitisation; digitisation and online accessibility of public domain material; digitisation and accessibility of in-copyright material; Europeana; digital preservation, taking into account the digital lifecycle approach to cultural material.

This evaluation is supported by an externally contracted study 7 (’evaluation study’), by various meetings and workshops with Member States, stakeholders and experts in the field, and by targeted and online public consultations.

The evaluation covers the whole period since the Recommendation’s adoption in 2011 to date. It covers the Member States, the European Economic Area countries (‘EEA’), Norway, Iceland, Lichtenstein, as well as the UK.

2.Background of the intervention

2.1. Description of the intervention and its objectives

The legal basis for the Recommendation is Article 292 of the Treaty on the Functioning of the European Union, which lays down powers for the European Commission to adopt recommendations in specific cases provided for by the Treaties.

At the time of its adoption, the Recommendation was an important action for one of the key areas tackled by the Digital Agenda for Europe, a flagship initiative in the Europe 2020 strategy 8 of March 2010. It built on the work done through the digital libraries initiative, with a view to making Europe’s cultural and scientific heritage accessible to all online. The launch of Europeana in 2008, Europe’s digital platform for cultural heritage, was one of the most important stepping stones for digital cultural heritage.

The Recommendation built on the 2006 Recommendation on the digitisation and online accessibility of cultural material and digital preservation (2006/585/EC) 9 , which had been an important means of supporting and encouraging the work of the Member States and CHIs in the field.

The Recommendation reflected the necessary changes in the policy context and in the cultural heritage sector since 2006, such as the establishment of Europeana, the recommendations stemming from 'the New Renaissance' report, prepared in 2011 by the Comité des Sages 10 , and the European Commission's proposal for an Orphan Works Directive in 2011.

Moreover, the Member States‘ reports on the implementation of the 2006 Recommendation for the reporting cycle covering 2008 to 2010 showed progress, but this was inconsistent and uneven for the different provisions of the 2006 Recommendation.

A modern and more effective set of measures for digitising and bringing cultural heritage online was needed to reflect the emerging new challenges and opportunities for the sector at that time. In particular, the Recommendation sought to address Europe’s untapped economic and cultural potential. Increasing digitisation and developing and applying advanced technologies to the sector simulates growth and job creation, as the process of digitisation is labour intensive and generates new jobs. Digitised cultural heritage resources are long-term economic assets which, through the data attached to them, create value and contribute to economic growth, with spillover effects into tourism, education, research, etc.

The Recommendation sought to address the need to better harmonise national approaches across Member States to the digitisation and preservation of cultural heritage. Considering the resources required for digitisation, a concerted action from the Member States was necessary to ensure more coherence in the selection of cultural material, better optimisation in the use of digitisation capacity and achieving economies of scale, as well as a more secure and stimulating climate for companies willing to invest in digitisation technologies. Stimulating innovative ways to make cultural and creative content available across the Member States results in more possibilities for users to explore and enjoy European cultural heritage treasures.

The Recommendation sought to address the need for increased digitisation, online access and preservation to avoid losing our cultural heritage assets and their memory. The increasing natural or human derived threats to historical buildings, museums and other cultural heritage assets also made it clear that, without digitisation and preservation of cultural heritage treasures, society would run the risk of losing its very ‘memory’ (digital and physical) and knowledge in the event of destruction or disappearance of those assets.

The ultimate impacts expected from the Recommendation were a better protection of Europe's cultural heritage, higher public awareness and appreciation of Europe's cultural heritage, increased availability of open data and exchange of knowledge.

The intervention logic of the EU action, in the form of a diagram summarising how the EU action was originally expected to work at the time of adoption/implementation, is provided below:

The Recommendation had the following main overall objectives: 

·optimise the cultural and economic potential of Europe’s cultural heritage

·ensure Europe’s position as a leading international player in the field of culture and creative content; and

·unlock investments from Member States and the private sector for digital cultural heritage initiatives

The main measures promoted in the Recommendation are:

1.Digitisation-organisation and funding: Member States are recommended to further develop their planning and monitoring of cultural material (targets and overviews), to encourage partnerships between cultural institutions and private sector, make use of the EU’s Structural Funds to co-finance digitisation activities, consider ways to optimise the use of digitisation capacity and achieve economies of scale;

2.Digitisation and online accessibility of public domain material: Member States are recommended to improve the access to and use of digitised cultural material in the public domain;

3. Digitisation and online accessibility of in-copyright material: improve conditions for the digitisation and online accessibility of in-copyright material;

4. Europeana: contribute to its further development;

5.Digital preservation: reinforce national strategies for the long-term preservation of digital material, update action plans implementing the strategies and exchange information with each other on the strategies and action plans; provide in their legislation for multiple copying and migration of digital cultural material for preservation purposes; make the necessary arrangements for the deposit of the material created in digital format.

Point 12 of the Recommendation encourages Member States to inform the Commission every 2 years of their implementation actions. The Commission monitors the progress made by the Member States in implementing the Recommendation with the help of the Expert Group on Digital Cultural Heritage and Europeana.

2.2. Baseline and points of comparison

2.2.1. Digitisation: organisation and funding

According to the evaluation study, when the Recommendation was adopted, the state of digitisation varied significantly depending on the type of institution and cultural material. While most CHIs had a digital collection or were involved in some digitisation activities (83%), only an estimated 20 % of all collections that needed to be digitised had in fact been digitised. Moreover, only one third of the institutions were part of a national digitisation strategy and only 34 % had a written digitisation strategy. Art museums were the most digitised (at 42%), while national libraries had digitised only 4% of their collections. Photographs were the most commonly digitised type of cultural material, followed by archival material.

Moreover, the Commission’s Second progress report on the digitisation and online accessibility of cultural material and on digital preservation in the European Union from 2010 11 , noted that, despite the important progress brought about by the 2006 Recommendation, the work carried out at the time to digitise cultural heritage content was still in its early stages. In particular, the report indicated that the vast majority of cultural objects worth digitising had not yet been digitised (e.g. for national libraries, only 3.5 % of relevant cultural objects had been digitised). While the national reports indicated that inventories, overviews and databases were increasingly being set up, to ensure a cross-sectoral and national dimension, several Member States had still not developed any national overviews of digitised collections The Commission’s report pointed to the fact that progress in digitisation activities depended on the long-term financing available, but very often a national budget had not been assigned to it and it was mainly the CHIs’ own resources that were financing digitisation (followed by project based financing with national and EU funds). Likewise, according to the report, some countries provided information about the sum of public budgets aggregated at national level specifically devoted to the digitisation of cultural heritage on an annual basis (e.g. France, Spain and Finland) or multiannual basis (e.g. Czechia and Belgium ).

Furthermore, the evaluation study 12 points to the limited adoption of policies on digital access by the institutions (around 31 % of them). Only 42 % of institutions reported that they were monitoring the usage of their digital collections through consultations or social media monitoring (libraries between 56 % and 78 %, museums around 27 %). The monitoring of use of their digital collections through web statistics was higher (~85 %). According to the Commission’s second progress report of 2010, mentioned above, 17 Member States had national coordination mechanisms for digitisation activities. As far as the targets were concerned, although there were quantitative targets in most of the Member States, they were usually set ‘bottom-up’ by the CHIs themselves and there was no comprehensive aggregated planning document at national level.

With regard to public private partnerships/PPPs (i.e. between CHIs and the private sector) although theoretically they could be a useful instrument to pool new resources for digitising cultural material, the national reports indicate they were not a widespread tool for financing digitisation.

Large-scale PPPs with both public and private partners (such as Google) had less difficulties in emerging than PPPs involving medium and small-scale cultural organisations.

2.2.2. Digitisation and online accessibility of public domain material and of in-copy right material

According to the Commission’s second progress report of 2010, only a minority of Member States (4) had mechanisms making it easier for cultural institutions to digitise orphan works and make them accessible online. Likewise, only a minority of countries (8) declared having mechanisms for out-of-print (out-of-commerce) works. Moreover, only three countries indicated having lists of orphan works and works in the public domain.

However, several of the countries took part in the EU-funded ARROW project ‘Accessible Registries of Rights Information and Orphan Works towards Europeana’, involving national libraries, publishers and collective management organisations.

2.2.3. Europeana

The Commission’s second progress report of 2010 pointed to a steady growth over time of access to digitised content through Europeana, reaching and significantly exceeding the original target of 10 million items by 2010 (i.e. more than 14 million items, including 5 million text items, were accessible through Europeana). However, despite an improvement in the overall balance, many Member States were not contributing a sufficient amount of content (i.e. about half of the Member States contributed in total less than 1 % of the total number of objects).

2.2.4. Digital preservation

The Commission’s second progress report of 2010 highlighted that the preservation of digital information remained a significant challenge, despite a growing acknowledgement by Member States of the costs linked to losing digital data. Almost half of them had adopted a strategy for digital preservation 13 . However, the diversity of the national strategies was ‘ranging from a preliminary feasibility study to a much more advanced and comprehensive digital preservation infrastructure with processes already up and running’ 14 . In addition, according to 'the New Renaissance' report of the Comité des Sages, referred to above, while mechanisms and plans had been put in place by several Member States, there was still a lot of work to be done to put all the strategies and plans into practice. This was the case not only in the Member States that were less advanced in this sector (and running the risk of permanent loss of digital content), but also in the Member States that were more advanced in this area.

According to the Commission’s 2010 progress report, 22 countries reported allowing multiple copying and migration of digital cultural material by public institutions for preservation purposes.

On policies and procedures for the deposit of material originally created in digital format, most Member States reported legal deposit policies in place for born-digital material, although some also reported actual challenges with implementation.

2.2.5. General aspects

From a more general perspective, whereas the evaluation study points to considerable room for improving the supply of digitised cultural material, on the demand side it acknowledges that there were few relevant and specific statistics on the use of digitised cultural material at that time.

Overall, the baseline shows that, at the time, despite the efforts and the progress of the Member States and CHIs to digitise, preserve and provide online access to digitised cultural heritage content, further steps were needed 15 .

3.Implementation / state of Play

3.1. Description of the current situation

The Recommendation represented a milestone in digital cultural policy, encouraging Member States to step up efforts and support CHIs with the digitisation, online access and preservation of cultural heritage.

3.1.1. Digitisation: organisation and funding

As mentioned in Section 2, the Recommendation encourages Member States to further develop their planning and monitoring of cultural material (targets and overviews), foster partnerships between cultural institutions and the private sector, make use of the EU’s Structural Funds to co-finance digitisation activities, optimise the use of digitisation capacity and achieve economies of scale.

According to the information provided by Member States in their implementation reports for the reporting period 2015-2017 and consolidated in the Commission progress report of 2018 16 , 23 of them have national digitisation plans and/or national funding programmes, as summarised in the table below.

Figure 1: Predominant trends in the organisation of digitisation across the EU

Source: Consolidated progress report on the implementation of Commission Recommendation, 2015-2017

The digitisation schemes have targets that vary considerably according to the Member State and that are usually set by each domain of cultural heritage (e.g. libraries, archives, museums, etc.). Digitisation of library and archival cultural resources is one of the main digitisation priorities in more than half of the Member States. In addition, for more than one third, immovable cultural heritage (e.g. monuments, historical buildings, archaeological sites) was a priority in the reporting period and these Member States allocated funding programmes to it. At least six Member States reported digitisation programmes for intangible culture (e.g. living arts, traditional folklore culture), while 3D appears to be gaining importance across the Union. However, as recalled also by contributions to the online consultation, only 22% of the heritage collections have been, on average, digitally reproduced 17 .

Discussions in the workshop on digital transformation in the cultural heritage sector, held in Lisbon in November 2019, pointed to the fact that the sector struggles to set priorities for digitisation and preservation and needs a clearer expression of the expectations and set benchmarks.

Moreover, according to the national reports, public-private partnerships followed a slow, upward trend, with mainly technology companies (e.g. Google) being interested in such schemes, followed by media publishers (e.g. Proquest) and banks. Some Member States (Czechia, Germany, the Netherlands, Austria, Italy) reported PPPs between national or state libraries and Google: in Czechia around 150 000 books had been digitised and made available since 2011 (when the PPP was established), while in Germany the PPP with Google led to the digitisation of around two million books. Wikimedia, international non-profit organisations and private individuals also concluded agreements with CHIs.

Two thirds of the Member States reported making use of the EU Structural Funds for digitising of cultural material and related services (Bulgaria, Czechia, Germany, Estonia, Greece, Spain, Croatia, Italy, Cyprus, Latvia, Lithuania, Malta, Poland, Romania, Slovenia, Slovakia, Finland and Sweden).

There are various ways to pool digitisation efforts to optimise the use of digitisation capacity and achieve economies of scale, ranging from competence centres and national or cross-border collaborations to sharing services and facilities. Eighteen Member States (Belgium, Bulgaria, Croatia, Cyprus, Czechia, Estonia, Finland, France, Germany, Hungary, Ireland, Latvia, Lithuania, the Netherlands, Poland, Slovakia, Spain, Sweden) reported competence centres, aiming to centralise funds and lower costs, consolidate the digitisation process, share best practices, etc. Moreover, two thirds of Member States reported sharing services and facilities (e.g. repositories, content management systems or IT tools) to store and manage digital objects and data more efficiently and make better quality content available online (Austria, Croatia, Denmark, Estonia, Finland, France, Greece, Italy, Latvia, Lithuania, Luxembourg, Malta, the Netherlands, Poland, Slovenia, Slovakia, Spain and Sweden).

A report on digitisation and IPR in European museums, 18 published in July 2020, shows that 43.6% of museum collections are digitised, although with different levels per museum category 19 . According to the participating museums, less than 20% of their collections are available online.

3.1.2 Public domain material and in-copyright material

The Recommendation encourages digitisation and online access to public domain and in-copyright material. The national reports pointed to a positive trend among Member States to ensure that public domain status is maintained after digitisation; most of them reported supporting actions ensuring wider access or use of the digitised cultural heritage material in the public domain.

Based on the information provided by the Member States in their national reports, the measures laid down in accordance with the Orphan Works Directive 20 do not appear to have contributed to large-scale digitisation of orphan works by CHIs. For the digitisation of out-of-commerce works, although a significant number of Member States reported licensing mechanisms supporting it, their practical application has been limited, due for instance to the fact that only some types of works (e.g. literary works) are concerned.

3.1.3. Development of Europeana

The Recommendation set the indicative target of having 30 million digital objects available on Europeana by 2015.

The quantitative targets for Europeana were already reached in 2013 21 and surpassed, (50 033 909 in July 2020), with images (27 945 338) and text (20 819 920) being the main categories of content 22 , compared with a marginal contribution of 3D for instance (16 662). Although the focus of the CHIs appears to be mainly the quantity, quality emerged as a need as well. More than a third of the Member States encourage their CHIs to submit high quality content and metadata to Europeana and 63.08% of the content is in tier 2 and above, as set by Europeana 23 .

A minority of Member States (Austria, Czechia, Germany, Greece, Estonia, Poland, Slovakia and Spain) reported making public funding for digitisation projects conditional upon making the digitised material available through Europeana.

Twenty Member States reported contributing to Europeana through national aggregators and through cross-border thematic and domain aggregators. Member States highlighted various challenges to aggregation, such as the need to strengthen the coordination capacity of national aggregators and to address the digital capacity of CHIs and more generally their digital transformation: for example by tackling difficulties with data submission and the lack of expertise in copyright legislation.

Around half of the Member States reported a wide and free availability of metadata of digitised cultural content. The reports also pointed to the need to further consider emerging technologies that can improve the quality of the digitised material.

Only a limited number of Member States (Austria, Cyprus, Greece, Luxembourg, Poland, Portugal and Spain) reported raising public awareness of Europeana through national communication actions or plans.

3.1.4. Digital preservation.

The national reports point to a mixed coordination approach at different levels to long-term preservation strategies and action plans, with most of the Member States reporting mainly strategies at the national level, specific activities by public institutions or bodies, and participation in European projects.

Almost half of the Member States and Lichtenstein reported explicit provisions, in their national copyright laws or laws on archiving, which allow multiple copying and migration of digital cultural material by public institutions for preservation purposes. This reporting includes amendments or new developments at the national level (i.e. in Czechia, Estonia, Germany, and Ireland).

Most of the Member States reported having in place measures ensuring that rights holders deliver works to legal deposit libraries and allowing web-harvesting techniques, while half of them have adopted legal provisions allowing the transfer of digital legal deposit works among legal deposit libraries.

4.Method

4.1. Short description of methodology:

The findings and conclusions of this report are based on a number of sources, including an evaluation study conducted by an external contractor, several consultation activities, desk research, various documents and exchanges with Member States and stakeholders.

The following main methodological steps were followed:

1.Inception phase:

This process implied organising the process, setting its objectives and the timeline and selecting the tools and the deliverables. In other words, the first phase focused on starting and agreeing on the conditions for the study (i.e. on the objectives and timing), carrying out an initial desk review and developing data collection tools. This step mainly involved the contractor and the Commission (which included the Interservice Group (ISG).

2.Data collection:

This process implied establishing the baseline of the analysis, carrying out national research, desk research and stakeholder consultations (including targeted and online public consultations) and organising workshops. This involved the contractor, various parts of the Commission, Member States’ competent authorities, cultural heritage institutions, international and umbrella organisations, the Europeana ecosystem of organisations and professionals, technology stakeholders with research, products and services in ICT for cultural heritage and the public.

The publication of a roadmap defining the scope of the evaluation gathered eight responses from various respondents.

The evaluation study was conducted by the contractor between August 2019 and March 2020 and updated in November 2020, following the online public consultation. The study was one of the main sources of information for this staff working document.

A consultation strategy was created and agreed with the ISG, with the following objectives: (1) to collect views on the Recommendation’s implementation and (2) to collect views on the impacts of possible future changes to the Recommendation.

As part of the consultation strategy, the Commission held five workshops with representatives of Member States, stakeholders, international organisations and networks in order to collect relevant and solid information on the Recommendation’s impact and on the challenges and needs confronting the sector, that have arisen from the digital transformation and that have to be addressed in the future.

Between November 2019 and January 2020, the Commission held a targeted consultation of Member States’ authorities with questions covering all five of the evaluation criteria. The Commission received 20 responses that fed into the evaluation study.

Between 22 June and 14 September 2020, the Commission held a twelve week open public consultation, focusing on the importance of digitisation and digital transformation in the field and how such processes can be supported, as well as on the Recommendation’s impact on the cultural heritage sector. The consultation gathered 565 responses. No significant limitations were encountered. More details are provided in the synopsis report in Annex 2.

An additional main source of information for the study and the evaluation were the national implementation reports by the Member States, which were consolidated in the Commission’s progress report (2018) on the Recommendation’s implementation (see Section 2 above).

3.Data analysis:

This implied assessing and compiling the results of research, surveys and consultations, evaluating and assessing findings and discussing them progressively at expert workshops. This involved the contractor, as well as the Commission (which included the ISG).

4.Synthesis and conclusions:

This implied drawing conclusions from the findings of the evaluation study, considering other inputs from stakeholders, and drawing conclusions on each evaluation criterion. This was mainly carried out by the Commission.

The final steps of this phase mainly consist of:

·assessing the outcomes of the evaluation study, of the consultation activities and of the implementation reports and then

·concluding on Recommendation’s impact in the cultural heritage sector across the EU and more particularly on its effectiveness, efficiency, coherence, relevance and EU added-value.

A more detailed description is provided in Annex 3.

4.2. Deviations from the evaluation roadmap

The evaluation roadmap published in July 2019 indicated that the evaluation would be completed during the third quarter of 2020. However, due to the COVID-19 pandemic and related priorities, the Commission opted to extend the evaluation timeframe until the end of 2020 to allow sufficient time to process and analyse the results.

4.3. Limitations and robustness of findings

Quantifying the costs and benefits of the intervention raised significant challenges. This was due, on the one hand, to the diffuse and intangible nature of the benefits of cultural heritage, and on the other, to the lack of available or updated data on costs in Member States. The lack of available or relevant data made it difficult to provide a monetarised analysis of the cost-benefits of the Recommendation and its impacts. For example, for cost data, the most recent figures on the costs of digitisation, date back to 2017. Moreover, the reports do not always provide relevant or consistent data, making it difficult to establish causal links and to draw proper conclusions. However, where quantitative data are missing, alternative proxy data were used and explicitly mentioned in the analysis.

The evaluation is considered robust and thorough.

5.Analysis and answers to the evaluation questions

The main findings of the evaluation, based on the five evaluation criteria, i.e. effectiveness, efficiency, relevance, coherence, EU added value, are summarised below and a list of the evaluation questions is provided in Annex 4.

5.1    Effectiveness

The assessment of effectiveness considers how successful EU action has been in achieving or making progress towards achieving its objectives. It looks particularly at: a) the extent to which Member States have implemented the Recommendation’s provisions, b) the extent to which the Recommendation’s objectives have been achieved and the achievements can be attributed to the EU intervention; what are the factors preventing these objectives from being achieved or that are helping them be achieved.

Key points on the baseline situation

ü The level of digitisation across the Member States was low and CHIs and a vast majority of cultural objects worth digitising were not digitised.

üDespite an increasing trend, many Member States contributed rather little to the content through Europeana.

üPPPs were not a widespread tool for financing digitisation.

Key findings on the current situation:

üThe Recommendation’s implementation by the Member States varies to a large extent.

üThe Recommendation led to i) an overall improvement in digitisation, online access and digital preservation, which was the result for instance of the adoption of national strategies and/or funding programmes for digitisation and domain-specific initiatives; ii) an increased number of PPPs; iii) more visibility and exchanges, leading to increased support and funding for cultural heritage; iv) practical measures to optimise the use of digitisation capacity, with positive trends since 2011, which can be attributed to the Recommendation.

üThe volume of digitised cultural heritage resources has increased, with the target levels in Europeana being exceeded ahead of schedule.

üThe overall impact of Europeana has been positive and acknowledged by those consulted for this evaluation and for the evaluation of the Europeana initiative the Commission carried out in September 2018 24 .

üHowever, the quantity of digitised cultural heritage should not be dissociated from quality; PPPs were concentrated in several countries, with smaller Member Sates experiencing more obstacles;

üImportant challenges related to copyright are expected to be addressed by the transposition and implementation of the Directive on copyright and related rights in the Digital Single Market 25 (2019/790/EU)

üThe Recommendation needs to be revised in order to address also the factors hindering the achievement of its objectives (e.g. lack of common standards and approaches, insufficient digital literacy and skills, insufficient funding) and to enhance its effectiveness.

The Recommendation has been effective overall. However, after 10 years, it has reached a ‘plateau’ in terms of its effectiveness and would benefit from a revision to increase its effectiveness. This is supported by the feedback from the online public consultation, where 41% of the respondents 26 considered that the Recommendation’s provisions have been implemented effectively and have achieved their objective to improve conditions in the areas addressed.

As shown in Section 3 above, the extent to which the various provisions of the Recommendation have been implemented 27  varies according to the Member State. This is the case for instance for the planning and monitoring of digitisation. In this respect, there has been a clear overall improvement resulting from i) the adoption of national strategies and/or funding programmes for digitisation, ii) domain-specific initiatives led by ministries, the use of regional schemes, and iv) quantitative targets at national, regional, institutional or programme level. For example, between 2011 and 2013, only 9 Member States reported using national strategies or funding programmes to digitise cultural heritage material 28 , whereas between 2015 and 2017, 23 Member States reporting doing so 29 . This increase can be attributed to the Recommendation. In particular, as acknowledged also during the meeting of the DCHE of November 2019 30 , the Recommendation stimulated discussions on digital for cultural heritage. It also raised awareness at all levels and across various sectors (e.g. finance), increasing the visibility of the domain, its needs and challenges, which led to more concrete support at national level (e.g. for digitisation projects).

During the DCHE meeting of November 2019, participants highlighted the effectiveness of the list of indicative targets for minimum content for each Member State’s contribution to Europeana, as laid down in Annex II to the Recommendation. The list represented an important incentive for the CHIs and Member States in their work to increase the quantity of digitised assets. However, stakeholders 31 also pointed out the fact that the Recommendation’s current provisions focus rather on increasing the quantity of the digital cultural material, and this might have triggered unexpected negative consequences for the quality. Achieving a high volume of digitised assets while ensuring high quality remains a challenge that needs to be addressed.

A key factor in supporting high-quality digitised cultural material is capacity building, with current capacity not responding sufficiently to the needs and challenges of the sector (see details below).

For the private investment through PPPs, the Commission’s consolidated progress report (2018) shows that the number of partnerships and institutions involved in PPPs has been gradually increasing across the EU, with technology companies such as Google and Proquest (for libraries), Family Search International (for archives) and Telefónica (for libraries and museums) continuing to be the main private partners for large-scale digitisation projects. Additional evidence 32 reveals PPP agreements with local software and technology companies in the field of cultural heritage mostly for digitisation, online access and digital preservation purposes. The evaluation study further points to specific challenges in the role and capacity of the PPPs in achieving the objectives of the Recommendation, challenges that were confirmed during the consultation activities. For instance, while smaller Member States, such as Estonia or Luxembourg, had difficulties in defining such partnerships, most PPPs tended to be concentrated in only some bigger Member States, such as France and Italy. Evidence 33 on the Netherlands, Hungary and Estonia confirms that they are limited and, as in the other Member States, predictably with technology companies and media publishers of educational materials. During the workshop held by the Commission in November 2019 in the context of the DCHE, stakeholders questioned the effectiveness of the provisions/conditions for public-private partnerships for digitisation, as laid down in Annex I to the Recommendation. Participants to the workshop on digital transformation in cultural heritage 34 underlined the significant role the private sector has in driving forward innovation and suggested that further collaboration should be sought to support digitisation efforts.

Since the Recommendation’s adoption, crowdfunding and match funding have emerged as alternative sources of funding for digital cultural heritage initiatives 35 . In 2016, for example, Film Archive Austria launched a campaign to restore an historical silent film, garnering support from over 700 contributors, who donated over EUR 86 000.

In case of European Structural Funds, 18 Member States reported using European Structural Funds to co-finance digitisation of cultural heritage during the 2014-2020 programming period 36 . According to the evaluation study incorporating contributions from the Member States 37 , Structural Funds played a key role in funding the digitisation of Europe’s cultural heritage in those countries where national sources were limited. The targeted consultation suggests that the Recommendation has been effective in helping to raise additional funds for digitisation, for example by giving the cultural heritage sector the opportunity to clearly convey the need for more support from policy makers.

When consulted, the representatives of the Member States in the DCHE also emphasised the Recommendation’s effectiveness in i) stimulating certain activities in their countries (e.g. for the development of a national digital library, digitisation projects), ii) prioritising digitisation over other matters and providing financial support, iii) enhancing coordination and drawing up digital strategies and iv) building up infrastructure at the national level.

However, stakeholders pointed to the fact that reliable funding sources for CHIs for the long term remain a challenge 38 . Findings from the Member State consultation survey indicates that while the Recommendation has generally been a useful instrument for promoting digitisation initiatives at national level, more work needs to be done to unlock long-term investments from ministries across Europe 39 (see section on Europeana below).

The Recommendation encouraged Member States to consider ways to maximise the use of digitisation capacity and achieve economies of scale. Member States reported that practical measures to optimise the use of digitisation capacity since 2011 have been positive, and this can be attributed to the Recommendation. Examples of such measures include the establishment of competence centres, the use of shared services or facilities, and national or cross-border collaborative measures to pool digitisation efforts and resources. One example 40 of innovative cross-border initiatives among CHIs is the Digital Heritage Network (DEN), previously referred to as the Netherlands Coalition for Digital Heritage. Overall, the Recommendation helped to develop closer cross-border cooperation and increase digital capacity in the cultural heritage sector. This has been confirmed by the representatives of the Member States in the DCHE 41 , who acknowledged the Recommendation’s effectiveness in facilitating the participation of CHIs in the Europeana network and its activities, contributing to a new mind set of openness and sharing.

Participants in the December 2019 workshop with international organisations and networks of December 2019 underlined that the Recommendation would be more effective if it further strengthened the various networks of CHIs and in particularcollaboration at international level (including by providing more support for the transfer of technology, the development of skills in non-EU countries).

However, feedback from the consultation activities point out that better coordination is still needed to ensure interoperability (including semantic) and standardisation 42 .

The Recommendation encouraged Member States to improve access and use of the digitised cultural material that is in the public domain, as well as the conditions for digitisation and online access of in-copyright material. The emergence of GLAM labs demonstrates how CHIs have made public domain material more accessible and reusable online 43 . GLAM labs promote innovative ways of working, bringing together institutions, technology, people and various communities. Examples include the International GLAM Labs Community, established in 2018, as a hub for international collaboration on digitisation and online accessibility. Moreover, at the workshop on digital transformation in cultural heritage in Brussels in January 2020, stakeholders acknowledged that the Recommendation had helped to trigger such initiatives.

However, findings from the workshops indicated that the Recommendation does not sufficiently address all the relevant copyright issues hindering the CHIs in promoting online access to and the use/reuse of digitised cultural heritage 44 ; CHIs requested an ecosystem aligning the rules of the Member States (i.e. with EU rules properly supported and enforced 45 ). Moreover, there is a general expectation 46 that the provisions of the Directive on copyright and related rights in the Digital Single Market will address some of the current obstacles 47 .

With regard to the Orphan Works Directive (2012/28/EU), by 2015 all the Member States had transposed the Directive. As pointed out in the evaluation study, Member States have promoted the availability of databases with rights information, connected at European level, such as ARROW and the Orphan Works Database, established by the European Union Intellectual Property Office, in accordance with the Orphan Works Directive. The Member States also contributed to the Framework for an EU-Wide Audiovisual Orphan Works Registry (FORWARD project), which ran from 2013 to 2017.

The current crisis made the need to improve online access more compelling, as acknowledged by stakeholders 48 . In particular, stakeholders pointed to various obstacles when accessing and reusing cultural heritage online, such as a lack of sufficient content, insufficient quality, the copyright and reuse status. They further highlighted 49 that obstacles to encouraging co-creation, co-curation and other forms of interaction persist. Likewise, collaboration with the archives is not sufficiently facilitated to promote an innovative work. Moreover, for both metadata and content data there is insufficient multilingual content and more needs to be done to promote this.

With respect to Europeana, the respondents to the consultation generally acknowledged that Europeana had a positive impact on the sector, with spillover effects in education (see below) and the creative industries.

As mentioned in Section 3, the indicative quantitative targets were achieved earlier than planned. However, while images and text were the main categories, there was only a marginal contribution of 3D digitised material despite the generally acknowledged significant potential and value of 3D digitisation technologies. Various initiatives promoted the use of 3D technologies within the sector. For instance, the 3D Content in Europeana Taskforce 50 was set up to identify ways to enhance the accessibility of 3D collections and Member States collaborated on several joint EU projects 51 . As mentioned above, additional efforts to promote quality content, including multilingual content, need to be made.

In terms of communication plans to raise awareness of Europeana among the general public and notably in schools, the Recommendation appears to have been less effective, as only seven Member States reported a national communication plan for this action 52 . Since the Recommendation’s adoption, various collaborative projects have taken place, such as the ones between the Europeana Foundation and the European Schoolnet, bringing digital cultural heritage to classrooms across Europe and providing further opportunities for digital teaching and learning 53 . However, feedback from consultations highlighted the need to further enhance teachers’ awareness of Europeana. Similarly, Europeana’s potential through CC0 licensing 54 should be better emphasised through collaboration with creative industries.

Using Europeana, and more particularly the Europeana Data Model, the Europeana Publishing Framework and the International Rights Statements to develop a standardisation approach in the cultural heritage sector has been important not only in the EU, but also at international level, where our models and best practices have been replicated.

However, as mentioned above, unlocking funding from ministries remains an important challenge overall. Figure 2 below indicates how the member States’ financial contribution to Europeana increased in 2011 (when the Recommendation was adopted), to sharply decrease afterwards.

Figure 2. Ministry funding for Europeana 2010-2018 (in EUR)

Source: Evaluation study (Europeana annual reports and accounts, 2010-2018)

The provisions on digital preservation encourage Member States to have appropriate and up-to-date measures for the digitised cultural material, while promoting coordination at EU level. As shown by the evaluation study, in 2014 only 26 % of CHIs had created a written digital preservation strategy that was endorsed by the organisation’s management, increasing to 42% by 2017. Stakeholders confirmed 56 the Recommendation’s contributions to increase digital preservation. Various initiatives have been taken since the Recommendation’s adoption, including the establishment of digital repositories and competence centres, the adoption of new software and technologies for digital preservation (e.g. the use of a repository of e-books on demand in Austria and the Nestor catalogue in Germany 57 ).

As shown in Section 3 above, there has been limited progress on multiple copying and migration, with half of the Member States having explicit legislative provisions to allow such activities by public institutions 58 .

Although there has been clear progress in coordinating approaches and preventing significant disparities across the EU in legal deposit arrangements, there is considerable scope for improvement 59 .

Most of the Member States currently have measures in place to allow preservation by using techniques collecting material from the internet, such as web-harvesting techniques, although for some this precedes the Recommendation’s adoption (e.g. in Denmark specific provisions were laid down through the 2005 Legal Deposit Law and the process is carried out by the Royal Danish Library, using the NetArchive web-harvesting platform 60 ).

Nonetheless, participants in the November 2019 DCHE workshop highlighted the need to strengthen and update the provisions on digital long term preservation, in order to meet the needs of the sector, in line with technology developments.

In order to be fully effective, the Recommendation should also clearly cover all the forms of cultural heritage not addressed at all or only minimally addressed, e.g. born digital, intangible (e.g. stories, dances or traditions, performances) or immovable. This is supported by the feedback from the online public consultation, where the majority of respondents (65%) were in favour of both broadening the scope of the Recommendation to include cultural heritage currently not addressed, and including the digital transformation of cultural heritage institutions. Moreover, digitisation of intangible cultural heritage has been considered very important by more than half of the overall responses (54%), while 37% considered it important. There is a convergence of views across the stakeholder groups, from 58% of NGOs considering it ‘very important’ to 47% of public authority respondents, but also a convergence of views across the sectors (e.g. tourism full agreement).

Based on the feedback from the consultation activities, and as illustrated above, the objectives of the Recommendation can be considered to have been achieved overall.

The evaluation study pointed to the following factors that have either contributed to or hindered the achievement of the objectives of the Recommendation:

a) Positive factors:

The Member States’ expert group, the DCHE, has been a useful forum for the Recommendation’s implementation and for the exchange of best practices among the Member States. It has also been acting as a governance body for the Europeana Digital Service Infrastructure, providing strategic orientations for its future and contributing to the sustainability of Europeana. More recently, the DCHE was also instrumental in getting 25 Member States to sign the 2019 Declaration of Cooperation on advancing the digitisation . This has also shown this group’s effectiveness for forward-looking initiatives.

Tools have been designed to support CHIs to become more confident about using digital to deliver on their missions, i.e. tools to design, organise and carry out digital cultural heritage activities 61 . The following examples can be mentioned: the Europeana Impact Playbook 62 , a step-by-step approach to help CHIs find inspiration, practical resources and networks to more efficiently achieve organisational goals; the Culture24 project 63 promoted training and upskilling programmes to drive up digital capabilities across the heritage sector, or the GIFT BOX 64 , a package of open-source tools and recommendations that help CHIs to make digital experiences richer for their visitors and to enhance engagement. The Europeana Publishing Guide 65 helps data partners to share collections in Europeana, bringing the existing information to one place and outlining the metadata and content requirements for publishing in Europeana.

Standardisation: Europeana has been particularly successful in the provision of standards regarding the publication and sharing of data that have been taken up in the cultural heritage industry across the EU and internationally, for example by the Digital Public Library of America (DPLA). This is, in particular, the case of the Europeana Data Model (EDM), the Europeana Publishing Framework (EPF) and the RightsStatements.org international initiative. The case study done for the evaluation of the Europeana initiative in 2018 highlighted that the Europeana Publishing Framework had a positive impact on the overall quality of data, although there was still a lot of scope for further improvement.

Technological advancements promoting open-source software and shared data have improved the online access of cultural heritage content and promoted collaboration between CHIs, with researchers, private or creative sectors, etc. Examples include specific factors like the FLOSS inventory for digital cultural heritage and digital humanities, listing all the Free Libre Open Source Software (FLOSS) relevant to the cultural heritage sector, or developments in cloud storage (helping CHIs maintain archives through digital preservation) and in linked data practices (promoting an open culture of online accessibility and reuse in the digital cultural heritage sector).

Opportunities, initiatives and collaboration provided by EU funding for the digital cultural heritage community (e.g. under the Framework Programmes for Research and Technological Development, Connecting Europe Facility or Horizon 2020) were a key factor that helped to achieve the Recommendation’s objectives.

b) Negative factors:

Lack of adoption of technical common standards and approaches at regional level for digitised content impeded the achievement of the Recommendation’s intended objectives (due, in particular, to legal constraints or incompatible file formats). The need for more interoperability of standards (including semantic) appears to still be a challenge for the sector, as stated in the workshops. Similarly, insufficient user-friendly multilingual services must be addressed. Participants in the workshops held 66 by the Commission highlighted the need to enhance the promotion and application of the FAIR principles, i.e. digital data should be findable, accessible, interoperable and reusable.

Insufficient advanced digital skills and literacy 67  in the cultural heritage sector are some of the key issues highlighted in the consultations. The digital shift has created new needs for the custodians of cultural heritage and new opportunities to be seized to support new forms of visitor experience and participation. Research for the evaluation study emphasised that insufficient advanced digital skills and literacy have an impact on the availability of the digitised content, on the efficiency of the process and on the possibilities for innovation 68 . As underlined also in the feedback from the online consultation, the need for a constant update of knowledge and skills in line with state-of-the-art technologies concerns all the digitisation related workflow and processes, from digitisation itself to access and preservation. Up-skilling staff with relevant and comprehensive digital learning mechanisms leads to enhanced digital skills and knowledge embedded in their work.

Insufficient funding for CHIs was also singled out as a major challenge by the respondents in the consultation activities 69 , in particular the difficulty to shift from project funding to stable sources of funding. CHIs, in particular the small ones 70 , struggle to secure a sustainable source of funding, and this has an impact on digitised content and innovation. The wide ranging impacts of the COVID-19 crisis underlined once more the need for the CHIs to get support for their activities, in numerous cases also to ensure their long-term survival (see Section below).

5.2. Efficiency

This section assesses the costs and benefits associated with the Recommendation’s implementation, more precisely the extent to which the Recommendation achieved its objectives at a proportionate cost.

Key point on the baseline:

üVery often a national budget was not allocated for digitisation activities.

üFor digitisation, CHIs mainly used their own resources (or national and EU funds for projects).

Key findings on the current situation:

üThe absence of available data to quantify the costs of digitisation and preservation triggered by the Recommendation’s implementation was a significant challenge, making it difficult to draw conclusions and make comparisons across Member States.

üEven if a more detailed analysis of the proportionality costs - benefits is difficult to carry out, stakeholders underlined that the benefits outweighed the costs.

üMost of the costs were borne by the Member States (with an uneven budget allocation across them) and by CHIs;

üThe Recommendation provided an opportunity for more discussions, exchanges and collaborations for the cultural heritage sector, resulting in more visibility and better prioritisation and thus triggering more funding.

üThe benefits were enjoyed by the whole of society: CHIs have benefitted from attracting and leveraging funding, collaborative networks and wider resources, expanding their audience; governments have benefitted from growth in cooperation and information sharing; creative sectors, teachers, students, the public in general have benefitted from a wider access to digitised cultural heritage resources.

üOverall, more than 900 cultural institutions across Europe have benefitted from EU support to make their collections accessible through Europeana, and to carry out targeted digitisation, while 50 033 909 cultural heritage items are digitally available through Europeana.

Respondents to the online public consultation gave positive feedback on the Recommendation’s efficiency, with around 70% of them either agreeing or strongly agreeing that the benefits of implementing the Recommendation are significant and justify the costs.

Most of the costs have been borne by a) Member States (governments) and b) CHIs, while benefits have been enjoyed by everybody/society as a whole (e.g. users, creative sectors, teachers, students).

The main limitation in this section is the absence of available data to quantify the costs of digitisation triggered by the Recommendation’s implementation. The reports do not always provide relevant or consistent data, making it difficult to establish causal links and to draw conclusions. Additional evidence 71 confirms this and highlights the fact that national support for digitisation emerges from a combination of programmes (not necessarily directly linked to the digitisation of collections) and it is thus challenging to track financial allocations for the digitisation of cultural heritage (along with the scope of the heritage field, the cultural policy structure, the coordinating structure).

Some of the respondents in the targeted consultation pointed to the challenges in quantifying the costs or barriers arising from the Recommendation.

a) Costs for the Member States:

Most of the evidence in this section relies on the budget allocated to digitisation (with budgets only as an indirect proxy for costs 72 ), together with qualitative analysis, i.e. information and views collected from the consultation activities organised by the Commission or in the context of the evaluation study. Moreover, as highlighted by the evaluation study, financial efforts for digitisation were already underway before the Recommendation’s adoption and they may have continued, but to what extent is uncertain. However, during the consultation activities, Member States and stakeholders generally acknowledged that the Recommendation provided an opportunity for the cultural heritage sector to engage in more discussions, exchanges and collaborations, resulting in more visibility and better prioritisation and thus triggering more funding.

The main resources allocated by the Member States to digitisation are presented in Figure 3.

Figure 3. Budgets (in EUR) and sources for digitisation of cultural material (2011-2017)

Source: Evaluation study.

While the figures presented above suggest that reporting of the allocated budgets is uneven, it is, however, clear that some Member States have been allocating substantial funds for digitisation activities. Larger Member States have been providing larger amounts and smaller Member States (in particular in southern and eastern Europe) have had smaller budgets and have been relying mainly on EU funds. For instance, in Germany, Saxony allocated EUR 2.5 million and Baden-Württemberg EUR 0.6 million.

However, the evaluation study highlights the limited and inconsistent data provided by the Member States over the period covering 2011-2017 (i.e. only 14 Member States), without a clear separation between national resources and EU funds. This make it difficult to draw conclusions and make comparisons across Member States.

b) Costs for CHIs

Most of the costs are borne by Member States (governments) and CHIs, the latter being responsible for digitisation. The costs of the CHIs have been used as a proxy for the costs stemming from the Recommendation, although it is not possible to establish causal links and a direct impact.

Feedback to from the online public consultation pointed to the fact that the cultural sector has been working with declining budgets despite an increase in technology related costs and with the pressure for more digitisation (and online access).

As shown by the evaluation study, based on the ENUMERATE surveys 90 , the average total costs 91 per institution doubled between 2012 and 2017, from EUR 176 000 to EUR 446 000 (Figure 4). Total costs grew depending on the type and size of the CHI. However, it is important to note that increasing the volume of digitised cultural heritage material resulted in higher costs.

Figure 4. Average total costs per institution for digital collections (in EUR thousand)

Source: Evaluation study

Evidence shows 92 that labour costs increased over time, with more full-time equivalents (FTE) in digital collection activities, from 5.5 FTE in 2012 to 11 FTE in 2017 (05). In terms of paid employment, the average FTE grew from 3.5 FTE to 7 FTE in the period. The increase in digitisation activities required, in addition to more full-time paid employees, volunteers, with an increase from two in 2012 to four in 2017. The growth in employment can be considered a positive effect of the Recommendation’s implementation.

Figure 5. Average FTE in digital collection activities per CHI, 2012-2017

Source: Evaluation study

Moreover, as shown by the evaluation study, between 2013 and 2017, the proportion of costs remained the same, with incidental costs (e.g. for the selection of material, the initial creation or acquisition of a digital collection) representing 53 % and structural costs (e.g. ongoing maintenance, enhancement and preservation of a digital collection) 47 % of the total costs.

In some Member States, costs are also borne by private sources. In the Netherlands, private funding is increasing 93 , although some sources are more for projects involving the digitisation and reuse of archival materials for publications, for example. In Estonia there are only a few of these private funds (e.g. the Soros Fund before closure).

Benefits of the Recommendation’s implementation: 

Despite the difficulty in quantifying such benefits, during the consultation activities stakeholders highlighted the benefits of the Recommendation for the whole of society.

a) Benefits for the Member States

As mentioned above, participants in the DCHE meeting in November 2019 noted that the Recommendation was important in promoting the sector and enhancing cooperation among the Member States, while fostering better organisation and coordination internally. This led to better prioritisation, allocation and a more efficient use of financial resources.

Moreover, raising public awareness of and through digital cultural heritage assets (e.g. see Section 2.2.2. on Europeana) has led to spill-over effects in other domains, e.g. tourism.

b) Benefits for the cultural heritage sector

As acknowledged during the consultation activities, the Recommendation’s implementation largely benefitted the sector 94 by exchanging information and more generally know-how, by promoting and sharing tools, networks and connections and by facilitating access to resources and digital activities that might not have been carried out without the Recommendation. It created additional opportunities to gain new perspectives from working with peers and experts in digital cultural heritage. This has been particularly relevant for smaller CHIs, whose access to funding and networks is more limited.

In this regard, the Europeana Network Association 95 has been very successful in bringing together heritage professionals, tech providers, educators, researchers and copyright specialists. Its six communities (Europeana Tech, Communicators, Copyright, Education, Impact and Research) allow numerous CHIs to network, learn and exchange best practices. Collaboration between the tech industry and the cultural heritage sector required joint forces, creating efficiency gains, stimulating innovation and enhancing the economic potential of Europe’s cultural heritage sector. Likewise, developing synergies with other public institutions stimulated progress on the use of new advanced technologies. For instance, the Tuscany Region in Italy and the University of Florence established the New Media for Cultural Heritage Competence Centre (NEMECH) 96 , connecting research centres and institutions, promoting the transfer of research know-how and providing digital technologies tools and applications for cultural heritage.

As mentioned above, increased visibility of the sector led to increased support for cultural heritage and access to more funding and infrastructure. Overall, more than 900 cultural institutions across Europe benefitted from EU support to make their collections accessible through Europeana, and to carry out targeted digitisation. This raised awareness among the cultural heritage sector on the importance of having a digital presence, has advanced the digital experience of cultural heritage institutions and mobilised cultural institutions to make their material accessible online. More than 3 700 institutions have contributed to Europeana and 50 033 909 European cultural heritage items are digitally accessible through it.

Also before the Recommendation, several countries did not have digital strategies, the Recommendation helped stimulate the adoption of such strategies and thus led to more funding becoming available for digital projects. Moreover, the greater number of discussions and exchanges of information have enabled cultural heritage custodians to better understand their needs and challenges in a rapidly evolving technological context. For instance, capacity building and particularly the upskilling of staff (not mentioned in the Recommendation) became an important element to consider in order to get prepared to efficiently and effectively use the opportunities provided by the digital transformation.

As pointed out by the evaluation study, although there is no direct or quantifiable evidence of the Recommendation’s impact on other sectors (e.g. tourism, education and research), the responses during the consultation activities highlighted the Recommendation’s contribution to the rest of the economy within and across Member States and the synergies with other projects and initiatives.

c) Benefits for the general public and society at large

As explained in the sections above, some of the benefits of the Recommendation, as mentioned during the consultation activities, include wider access to previously restricted content and democratisation of culture and knowledge, grater efforts to innovate and develop new technologies and services with spillover effects ino other sectors (e.g. tourism, research, education).

Moreover, the Recommendation stimulated policies, strategies and projects at EU, national and local levels for the benefit of the whole of society; it encouraged co-creation and co-curation, while keeping people and communities connected. Historiana is a good example of an initiative promoting virtual learning for educational purposes; its collaboration with Europeana aimed to create new ways for history educators to create, share and use e-learning activities with content from the Europeana collections website 98 .

Responses to the consultations indicated that improving digital skills and digital literacy leads to further efficiency gains, as mentioned above. The Recommendation should take into account how digital content is produced and used by cultural heritage professionals, creative sectors, research, education, users, etc. Up-skilling staff with relevant and comprehensive digital learning abilities leads to enhanced digital skills and knowledge embedded in their work. Doing sowould lower costs and maximise resources. Similarly, increased collaboration and common initiatives, sharing of know-how and networks would lead to a more efficient use of resources and the fulfilment of organisational missions.

Figure 6. Comparison of benefits and costs

Legend: the '+' indicates the Recommendation’s impact, ranging from small (+), medium (++), and large (+++).

Source: Evaluation study

During the consultation activities respondents highlighted that digital transformation is not an option anymore, it is a reality, it is a requirement of the current technological development of society and CHIs must embrace and support it. This may translate (initially) into additional costs (e.g. in infrastructure, in digital skills and literacy), but also benefits (e.g. more agile processes, improved knowledge and visibility, reaching wider audiences). Results can be achieved with fewer resources only once the required changes have been implemented and the processes are optimised and working. Increased collaboration between private and public, as well as between cultural heritage professionals, will lead to efficiency gains, while greater synergies with other sectors will benefit the whole of society. 

Overall, it was acknowledged that implementing the right digital tools and acquiring the necessary skillsets can help cultural heritage professionals to optimise processes and resources, while building confident CHIs and engaged users.

However, in a context where the COVID-19 crisis strongly hit the sector, threatening the long-term financial sustainability of many CHIs, funding becomes even more important in a context where governments will most likely make additional budgetary cuts 99 .

5. 3. Relevance

The main question addressed in this section is whether and to what extent the Recommendation has been and is still relevant to the needs of the stakeholders. This section looks at how well the Recommendation is adapted to future technological and social advances.

Key points on the baseline:

üDespite the efforts and progress of the Member States and CHIs to digitise, preserve and provide online access to digitised cultural heritage content, further steps were needed.

üThe need for more digitisation, online access and preservation, along with the need for more harmonised efforts across the EU to unlock Europe’s economic and cultural potential made the case for the Recommendation.

Key findings on the current situation:

üStakeholders generally acknowledged the Recommendation’s relevance in responding to the needs identified at the time of its adoption;

üEvidence pointed not only to the Recommendation’s relevance, but also to the need to strengthen it to adapt it to the current needs and challenges.

üSocio-economic changes, technological advancements, legislative changes create a different context for CHIs that must be taken into account. The Recommendation needs to better reflect the huge potential of advanced digital technologies (e.g.3D digitisation) and emerging technologies (including immersive technologies such as VR or AR and AI), bringing unprecedented opportunities for digitisation, online access and preservation.

üThe need for advanced digital literacy and skills is one of the key issues highlighted in all the consultation activities.

ü It is important to make full use of cultural heritage data for the benefit of the sector and to enhance interoperability and capacity building.

üIntangible and born-digital cultural heritage need to be taken into account in any future revision of the Recommendation.

The feedback gathered from the consultation activities 100 revealed a general acknowledgement of the relevance of the Recommendation in responding to the needs identified at the time of its adoption.

As mentioned above, the Recommendation sought to address the need to a) unlock Europe’s economic and cultural potential, b) better harmonise national approaches across Member States to digitisation and the preservation of cultural heritage, as well as c) increase digitisation, online access and preservation to avoid losing our cultural heritage assets and their memory (due to floods, fires, mass tourism, etc.).

At the time of the Recommendation’s adoption, the context for digitisation efforts was changing: more people were consuming culture online, CHIs needed to change tools and organisational culture and the creative sectors, which are a driver for innovation and economic growth, were flourishing. Managing the ‘digital shift’ required a strategic approach defining how creativity could be better promoted and how access to and the use of digitised cultural resources could be facilitated, while preserving them for the next generations 101 .

Stakeholders and Member States representatives acknowledged the Recommendation’s contribution in encouraging public authorities to prioritise and increase support for digitisation and provide more funding, while increasing online access and preservation. Using digital for creation and culture led to unlocking the potential for broader societal, cohesive and economic benefits from sectors such as sustainable tourism, education and creative sectors. The impact was thus on the whole of society: on CHIs (e.g. able to reach out to broader audiences and engage with new users), creative sectors (e.g. stimulation of creativity and content), and teachers, students, the public (e.g. wider access to cultural material), etc.

As mentioned in Section 5.1, participants in the DCHE workshop of November 2019 highlighted the importance of the Recommendation in helping CHIs to participate in the Europeana network and its activities, thus encouraging the sharing of knowledge and best practices. Through Europeana, the Recommendation has also played a significant role in the digital development of the cultural heritage sector in Europe by strengthening cooperation and standardising cross-border activities. 

Moreover, the Recommendation’s relevance through Europeana and more particularly through the synergies created within the sector of education, i.e. to provide further opportunities for digital teaching and learning (see Section 5.1.) has been confirmed by the responses to the 2017 Eurobarometer on cultural heritage 102 , where a large majority of respondents (88%) agree that Europe's cultural heritage should be taught in schools.

As indicated in Section 2 above, evidence points to the high number of objects that were not digitised 105 before the Recommendation and to the key role it had in promoting and supporting digitisation and online access to digitised content. This further helped to avoid the complete loss of the memory of cultural heritage assets due to natural or human derived factors.

While the outcome of the consultation activities revealed a general acknowledgement of Recommendation’s the relevance in responding to the relevant needs and achieving its objectives, the consultations also revealed a strong need to strengthen the Recommendation, to make it better reflect the current technological, economic and societal state of play. More particularly, 95% of the respondents to the online public consultation agreed that the Recommendation should be updated to reflect today’s needs in the area and increase the potential of Europe’s cultural heritage.

Since the Recommendation’s adoption, there have been significant and interlinked changes in the context in which digitisation, online access and digital preservation take place: socio-economic changes leading to different and increased user needs, technological changes (e.g. AI, machine learning), legislative changes (e.g. on copyright). These changes resulted in new opportunities and challenges for the CHIs and guided their choices and actions. The majority of the responses from the DCHE representatives in the targeted consultation pointed to the fast paced changes in the digital environment that the sector needs to cope with and urged the Commission to revise the Recommendation to keep it relevant for the future.

As emphasised by the evaluation study, better access to and a greater use of the internet, together with increased consumption of online goods and services, higher cultural employment and government expenditure on recreation, religion and culture are all examples of what has helped to create a new environment for both providers and users of Europe’s digitised cultural heritage. The development of these factors is shown in Figure 6.

Figure 6. User characteristics in relation to internet use and the culture sector, EU-28 average 106 , 2011 - 2017/19

Source: Evaluation study

The 2017 Eurobarometer 113 reveals that over half of the respondents (55%) have used the internet in the last 12 months for at least one of a range of cultural heritage purposes, and a large part of them (31%) for the accessibility, facilities and main features of a museum, historical monument or traditional event in preparation for a visit or their holidays. In addition, almost one quarter (23%) used the internet for buying or booking services for events or activities (e.g. tickets, guided tours, etc.) while 21% used it for viewing cultural heritage-related content, such as the description of a work of art or historical monument during a visit, historical information about a traditional event they attend; 11% of the respondents have created or shared cultural heritage-related content, such as a picture or a video of a work of art or historical monument, etc. This shows a high interest of Europeans in getting a wider virtual access to cultural heritage and confirms the Recommendation’s objectives and relevance.

The responses to the online public consultation highlighted the sector’s efforts to increase online access during confinement, through more digital learning resources (e.g. recording lectures and support material to be streamed); more digital activities (such as extending online access to curated audio-visual assets, virtual tours of exhibitions), increased social media presence and online collaboration. The COVID-19 pandemic has acted like a catalyst for increased online activities and services, with the majority of respondents stating that they envision additional changes by their organisations in the future (e.g. better procedures, better quality of the digital content, more frequent use of collaboration platforms especially for teaching and research activities). However, if on the one hand the crisis accelerated the digital transformation of the heritage sector, on the other it also deepened the current inequalities between large and small CHIs (i.e. the latter not having the same resources to share their collections and materials online, missing out on digital opportunities) 114 .

The development of digital technologies such as AI, computer vision, deep learning, machine learning, cloud computing, Big Data, VR and AR has brought unprecedented opportunities for digitisation, online access and preservation. Digital technologies can empower and encourage people to participate in culture in a more active and creative way 115 . In particular, AI allows manifold uses of diversified and rich data across Member States and CHIs. For instance, it has a great potential to extract knowledge from different data sets of cultural artefacts held by CHIs across Member States. AI can allow users to navigate through extremely vast amounts of cultural and creative content 116 , while reducing the efforts and resources required for such effort by CHIs. AI can also boost automated annotation of digitised cultural heritage, or can help to reconstruct lost cultural heritage information or to extract new knowledge from cultural heritage data 117 .

As underlined in the feedback from the public consultation 118 , VR/AR technologies can enhance the user experience, while providing great potential for use in education and research.

3D technologies are now mature enough to help in preservation and restoration or in providing virtual access to cultural heritage where it is difficult or impossible to get (e.g. under water). These technologies provide numerous opportunities for CHIs to reach wider audiences, with more immersive experiences, for both onsite and online visitors. The need to focus more on higher quality and more particularly on 3D emerged from both the consultation activities, and from the reports of the Member States 119 . For instance, a large majority of overall respondents to the public online consultation consider it very valuable or valuable to create digital twins of cultural heritage buildings, monuments and sites such as Notre Dame, ahead of any damage by fire or other disasters and more than half of the respondents (53%) agreed that 3D technologies are very valuable for creating high-quality 3D models of museum objects (in addition to 38% agreeing it is valuable). There is a clear need to strike the right balance between quantity and quality, moving away from the current approach focused mostly on quantity. In this respect, stakeholders argued 120 that since the Recommendation’s adoption, the emphasis has been on increasing the volume of digitised cultural assets and ensuring a high quality for all of them (including metadata) and this remains a challenge.

Therefore, in order to remain relevant, the Recommendation should address the role advanced technologies can play in the field (e.g. strengthening the ways CHIs could deliver higher quality content 121 , supporting new forms of visitor experience and engagement), while fostering inclusivity, equality, diversity. It is important to note that a large majority of the respondents to the public consultation (and also across the sectors) agreed that, based on the lessons learned from the COVID-19 crisis, digital technologies would be very important in making their organisation more resilient in the future while helping to increase the number of people accessing cultural heritage through the internet. 

Participants in the workshop on digital transformation held in Brussels (2020) indicated the need to properly address the issue of making full use of cultural heritage data for the benefit of the sector. Cultural heritage data are currently not sufficiently collected, used and reused within this sector or across other sectors (e.g. education, research) and the Recommendation would need to properly reflect this. Social media and the existing web data could, for example, be an important source of data and more efforts should be placed on this. In terms of repositories or other elements of data infrastructure, maximising their impact through a wide scale initiative is important 122 . The respondents to the online public consultation (96%), considered common data spaces for sharing cultural heritage data and building high-value data sets to be valuable instruments in the field. This view was particularly shared by stakeholders such as academic or research institutions, public institutions, the public or sectors such as the technology /ICT, cultural, education and research sectors. Likewise, respondents to the online consultation considered repository infrastructure for digitised cultural heritage content (95%), cloud infrastructure (90%), supercomputing capabilities for large-scale cultural heritage simulations (76%) key to cultural heritage.

While the Recommendation has contributed to more interoperability in the sector by supporting the development of a framework of interoperability and increasing awareness of common frameworks, there is still work to be done for instance on semantic interoperability or interoperability of standards 123 .

Ensuring appropriate means for preservation implies not only sustaining single digital objects, but also creating the right context for them to be understood in the future. Overcoming technology obsolescence is a key element in avoiding a complete or partial loss of digital objects and is still a complex task 124 .

Users have become more demanding with regard to the products and services they consume/use, wanting to participate, enjoy and learn more. They challenge the CHIs more and require a stronger link with them, i.e. to maximise and diversify the possibilities to digitally interact with cultural content.

According to stakeholders 125 , in order to embrace digital transformation, a key factor that professionals of the sector need to address is capacity building. In this respect, providing staff with advanced digital skills and literacy remains a big challenge for many of them. In particular, half of the respondents to the public online consultation confirmed that their organisation had struggled with the advanced digital skills needed to manage their online presence during the pandemic. Respondents working in the education and research sectors were likely to be the most affected, followed by respondents from the culture sector. Additional evidence 126 collected during the pandemic confirms that many CHIs in rural areas did not have the necessary digital skills to carry out their activities during the lockdown, leading to a significant void in culture on offer. 

In addition to the large variety of CHIs and the many different tasks that staff need to carry out, knowledge and skills should be constantly updated and kept in line with technological developments. Feedback from stakeholders, particularly from the January 2020 workshop 127 pointed to the importance of having a common standard and classification of skillsets for cultural heritage professionals (e.g. for senior and middle management, as well as expert levels), depending on the specific needs and domains (e.g. legal, technical, life-cycle management, communication). Other stakeholders pointed out for example that researchers and practitioners do not have the knowledge and ability to master digital tools and devices to deal with the accelerated development of ICT 128 . Ensuring the necessary (specific) skills and capacities of the professionals of the sector is a priority for the creation, management and development of digitised cultural heritage, with spillover effects ino other sectors. Feedback from the online consultation highlights what stakeholders consider to be the core set of digital skills that professionals from the cultural heritage sector should acquire: data and metadata management and analytics, and copyright and licensing, followed by digitisation (2D and/or 3D) 129 . Furthermore, findings from the DCHE workshop in November 2019 highlighted the importance of a higher competence level for staff that would drive up capabilities across the heritage sector.

At a more general level, in the consultation on the strategic objectives of the Digital Europe programme, held between 25July and 25October 2019 130 , 74% of the respondents indicated that the EU should support the Member States in improving the advanced digital skills necessary to operate the up-to-date digital infrastructures. Likewise, a large number of responses considered EU support for culture very important.

New forms of cultural heritage have emerged, and their collection and preservation remain a challenge, as indicated by stakeholders during the consultation activities (e.g. for born-digital heritage, meaning it has been created digitally, without an analogue equivalent or cultural content created outside CHIs, such as on social media or the gaming industry 131 ). Due to a rapidly evolving technology, with hardware and software becoming quickly obsolete, born-digital cultural heritage needs to be properly collected, managed and preserved to be accessible and usable in the long run.

While in the short term, most of the CHIs have been focusing on the immediate challenges raised by the COVID-19 pandemic, new models and ways of working will emerge in the longer term 132 . CHIs will learn from the pandemic, by creating innovative approaches for digital literacy and skills, processes, content and communication, through a new organisational culture and mindset.

5. 4. Coherence

This section assesses how the various actions have worked together to achieve the objectives. It looks particularly at the extent to which the Recommendation is aligned and consistent with other EU policies, with Europeana- related policy/initiatives and with cultural heritage policies in the Member States.

Key points on the baseline:

üThe Recommendation has created synergies with the European Agenda for Culture (2007), helping to shape the policy framework for the cultural sector.

üIt has built on the work done through Europeana (2008).

üAt the time of its adoption, the Recommendation was an important action for one of the key areas tackled by the Digital Agenda for Europe, a flagship initiative of the Europe 2020 strategy (2010).

Key findings on the current situation:

üOverall, the Recommendation is aligned with several other policy interventions in different fields, such as culture or copyright. In particular, evidence suggests that the Recommendation is consistent with the New European Agenda for Culture, the Work Plan for Culture covering the period 2019-2022, the European Framework for Action on Cultural Heritage, the Digital Agenda for Europe, the Digital Single Market strategy, the Directive on copyright and related rights in the Digital Single Market and the Directive on open data and the reuse of public sector information.

üHowever, a future revision of the Recommendation should aim to ensure better alignment with these more recent initiatives such as the new digital strategy ‘Shaping Europe’s Digital Future’ to create a Europe fit for the digital age and with ’A European strategy for data’.

A majority of the respondents to the online public consultation (around 59%) considered that the Recommendation had been coherent with other EU policies and initiatives.

The Recommendation is consistent with several other EU policy interventions in various fields, such as culture and copyright, or is aiming, more generally, to unlock Europe's innovative capabilities, and to exploit the economic and societal benefits of a digital society.

In the field of culture, the Recommendation has been sharing the same objectives and efforts with more specific actions in the field, to promote culture and its key role in Europe’s society and economy. For instance, the Recommendation has created synergies with the European Agenda for Culture, adopted in 2007 133 , helping to shape the policy framework for the cultural sector. By encouraging the Member States to take actions for greater digitisation, online accessibility and preservation, the Recommendation has contributed to the objectives of the European Agenda for Culture, namely:

·promotion of cultural diversity and intercultural dialogue;

·promotion of culture as a catalyst for creativity in the framework of the Lisbon Strategy for growth and jobs;

·promotion of culture as a vital element in the Union's international relations.

The Recommendation was an important action for the Europe 2020 strategy of March 2010, and in particular for its flagship initiative ’A Digital Agenda for Europe’, aiming to deliver sustainable economic and social benefits through a Digital Single Market. Providing active support for the digitisation of Europe's rich cultural heritage was one of its commitments, and it translated, among other things, into the support provided by the Recommendation. It built on the work of one of the stepping stones of digital for cultural heritage, i.e. Europeana, launched in 2008.

As highlighted in the Commission’s Communication ‘Towards an integrated approach to cultural heritage for Europe’ 134 , digital tools can contribute to better public access to different forms of cultural and linguistic content, while facilitating their preservation.

However, to be more consistent with the EU policy on culture, the Recommendation should further address key aspects for cultural heritage, as underlined by the Communication mentioned above, for instance the need to:

·encourage the modernisation of the heritage sector, raising awareness and engaging new audiences ;

·seize the opportunities offered by digitisation; to reach out to new audiences and engage young people in particular;

·identify skills needs and improve the training of heritage professionals and

·continue developing more participative interpretation and governance models that are better suited to contemporary Europe, through greater involvement of the private sector and civil society.

The Recommendation is consistent with the New European Agenda for Culture 135 adopted in 2018, sharing the same objectives of the Agenda’s three dimensions, namely:

·Social dimension - harnessing the power of culture and cultural diversity for social cohesion and well-being. For instance, enhanced digitisation and its wider access contributed to the protection and promotion of Europe's cultural heritage as a shared resource. Particularly through Europeana and its resources and initiatives, it contributes to raising awareness of our common history and values and reinforce a sense of common European identity 136 .

·Economic dimension - supporting culture-based creativity in education and innovation, and for jobs and growth. For instance, through Europeana (see Section 5.1.on effectiveness) it has provided further opportunities for digital teaching and learning, promoting arts, culture and creative thinking in education and lifelong learning.

·External dimension - strengthening international cultural relations particularly through Europeana and the collaboration with other countries. Europeana is, for instance, still the only initiative of its kind with the largest digital collections in the world offering material in almost 40 languages.

However, in order to be more aligned with the New Agenda, the Recommendation should, for example, further promote digital skills needed by the sector, strengthen the innovation capacity of creative sectors through a wider access to digitised resources.

Moreover, the Recommendation has synergies with the Council Work Plan for Culture for 2019-2022 137 . In line with the strategic framework of the new Agenda for Culture, this Council work plan acknowledges major developments such as the digital shift and the changing working environments, as well as the need to deal with these opportunities and challenges through joint efforts in cultural policy.

The Recommendation is closely aligned with European Framework for Action on Cultural Heritage, proposed for the first time in 2019, to capture and scale-up the success of the European Year of Cultural Heritage 2018 138 . For instance, the Recommendation shares specific objectives with the Framework, e.g. engaging the wider public and removing barriers to access to cultural content. However, while the Framework considers the tangible, intangible and digital dimensions of cultural heritage as inseparable and interconnected, the Recommendation would need to be further strengthened to address also intangible and digital-born cultural heritage.

The Recommendation is aligned with the Digital Agenda for Europe 139 and its successor the Digital Single Market strategy 140 , which highlight the opportunities brought by technological advancement to address societal challenges, i.e. digitising Europe's cultural heritage and making it available to society at large. The Recommendation’s purpose is to enhance digitisation and to make digital content widely available, while supporting preservation. Both strategies also pointed to the lack of digital literacy and skills, which remains a challenge for the cultural sector that was widely acknowledged during the consultation activities and that the Recommendation currently does not address. However, the Recommendation would need to be aligned with the priorities of the new digital strategy ‘Shaping Europe’s Digital Future’ 141 to create a Europe fit for the digital age and with the‘European strategy for data 142 ’ adopted in February 2020, which calls for appropriate policies and investments to pool European data in key sectors, with EU-wide common and interoperable data spaces.

The Recommendation is closely aligned with the Declaration of cooperation on advancing digitisation of cultural heritage 143 signed by 27 European countries and aimed to encourage a better use of state-of-the-art digital technologies for cultural heritage, enhancing its use and visibility, while improving public engagement. The synergies between the Recommendation and the Declaration concern for instance the reuse of digitised cultural resources to promote citizen engagement, innovative use and spillover to other sectors, with closer cross-border cooperation. However, while the Declaration recognises capacity building as one of the key challenges that has to be addressed, the Recommendation would need further alignment in this respect.

The Directive on copyright and related rights in the Digital Single Market 144  modernises the copyright framework and lays down provisions addressing the way CHIs operate in the digital environment. The Recommendation is coherent with the Directive, sharing some of its main objectives, such as increased cross-border access to content online and wider opportunities to use copyrighted material in education, research and cultural heritage. The new provisions of the Directive should facilitate CHIs digitisation activities, use and reuse of digitised resources through a legal mechanism to facilitate the conclusion of collective licensing agreements for the digitisation and dissemination of out-of-commerce works (e.g. books that publishers have stopped printing and selling) held by CHIs in their permanent collections. The origin of the out-of-commerce works provisions of the new Copyright Directive goes back to and expands on the non-binding 2011 Memorandum of Understanding on Key Principles on the Digitisation and Making Available of Out-of-Commerce Works 145 , which the Recommendation directly refers to.

Participants in the Workshop on Digital Transformation in Cultural Heritage 146 , along with feedback on the roadmap, indicated that the Recommendation does not sufficiently address certain copyright issues that may prevent CHIs from promoting online access, use and reuse of digitised content. However, it is expected that the Directive’s transposition and implementation will address important copyright challenges.

Likewise, the Recommendation is aligned with the Orphan’s Works Directive, sharing common objectives to improve digitisation and online accessibility of in-copyright material and encourage its transposition and implementation.

The Recommendation shares synergies with Directive (EU) 2019/1024 on open data and the reuse of public sector information 147 whose transposition and implementation should remove remaining barriers and cater for a more efficient use and reuse of digital cultural resources.

The Europeana strategy for 2015-2020 focused on improving data quality, opening the data and making it available without restriction for reuse, thus creating value for partners. The provisions of the Recommendation encourage the achievement of similar/close objectives as well, bringing the Recommendation in line with the strategy. The main priority of the new Europeana strategy for 2020-2025 is to enhance CHI’s capacity to seize the opportunities and how best to respond to the numerous challenges opened by digital transformation. As suggested in most of the consultation activities, the Recommendation should be revised in order to better support CHIs to make the most of digital technologies to record, preserve, document and share cultural heritage online. This implies addressing in a more effective and efficient manner emerging needs, such as the need to build capacity. Indeed, advancing digital literacy and skills has emerged as a recurrent topic raised by the professionals of the sector, while underlining the importance of a network of supporting competence centres. Moreover, a stronger emphasis on the quality of data and metadata, by using new/advanced technologies like machine-learning, would increase coherence with the Europeana strategy for 2020-2025.

Feedback from the representatives of the Member States to the DCHE underlines the Recommendation’s coherence with and complementarity to cultural heritage policies in the Member States. The various actions carried out by the Member States as a follow-up to the Recommendation confirm the coherence between the Recommendation and their national policies. The digitisation policies put in place by the Member States vary to a large extent according to their priorities, resources and coordination mechanisms. However, although there is no size fits all, for the 2015-2017 reporting period, most Member States had national strategies/plans and national funding programmes in place, which corresponds to the Recommendation’s priorities. Likewise, the digitisation of library and archival documents was one of the main digitisation priorities, including manuscripts, books, journals and historical newspapers, underlining the coherence between the Recommendation and the national policies in the field.

As highlighted in the Europeana evaluation 148 , the initiative is in close alignment with Member States’ national policies on digitisation and the online accessibility of cultural heritage material. 90% of the respondents indicated some degree of complementarity between Europeana and national initiatives (such as Gallica, Hispana or the German Digital Library), highlighting in particular Europeana’s role as the only organisation that provides access to cultural material across borders in Europe. Moreover, 61 % also indicated some degree of overlap with these. As shown by the national reports, Member States have stepped up their support to the Europeana ecosystem, through national and regional aggregators; national and regional aggregation initiatives have increased in importance at national level and for Europeana, while participation in European aggregation projects has expanded 149 .

5.5. EU ADDED VALUE

This section assesses the benefits stemming from the intervention, more particularly the extent to which it a) generated an increase in the digitisation of and online access to cultural material, as well as in the actions taken for digital preservation (beyond what Member States would have achieved alone); b) led Member States to broaden the types of measures implemented to promote the digitisation of and online access to cultural material; c) promoted innovation and the transfer of ideas across Member States in this field.

Key points on the baseline:

üEfforts to digitise cultural heritage content were in their initial stages.

üIt was mainly due to the CHIs ’own resources for financing digitisation, as very often governments did not allocate a national budget/resources.

 Key findings on the current situation:

üThe Recommendation had a clear added value in increasing digitisation (e.g. quantitative targets were exceeded) and online access to cultural heritage resources, as acknowledged in the context of the consultation activities.

üIt significantly contributed to raise awareness within and beyond the sector, in promoting national/regional and cross-border collaboration, exchanges of experience, knowledge, best practice and useful tools among the Member States.

üThe Recommendation provided a forum for discussions and cooperation between Member States and between CHIs, leading to increased visibility and funding opportunities for the latter.

A large majority of the respondents (80%) to the online public consultation provided a positive feedback about the added-value of the Recommendation compared to initiatives that Member States would have taken in the absence of it.

As shown in the previous sections, there was a clear increase, supported by the Recommendation, in the digitisation of and access to digitised cultural resources. The quantitative targets encouraged by the Recommendation were surpassed earlier than planned. The Council Conclusions of 31 May 2016 highlighted not only the progress towards the digitisation, online accessibility and digital preservation of cultural heritage, but also the joint efforts of CHIs, Member States and the Commission 150 . This would have been less likely without the Recommendation 151 .

The Recommendation had a significant contribution in raising awareness within and beyond the sector, in strengthening exchanges of experience, knowledge, best practice and tools among the Member States. It urged them to take action and track progress in order to increase the digitisation and online accessibility of European cultural heritage and to improve their national policies/strategies or strategic planning. For instance, it was an important source of inspiration for the development of the National Digital Library in Finland. It stimulated greater harmonisation and deployment of standards, better interoperability and preservation requirements.

Furthermore, involving the private sector in digitising cultural material led to successful and innovative partnerships. For example, in Austria, the digitisation of ‘Cinema in the Austro-Hungarian Monarchy’ was supported in 2016 by the private cultural institution the Klimt-Foundation 152 .

The provisions of the Recommendations on optimising the use of digitisation capacity and achieving economies of scale appeared to be important for all Member States and to be applied to a large extent. Joint efforts for national or cross-border collaborations competence centres or sharing services and facilities such as repositories or IT tools were deployed in order to make digitisation related activities more effective and efficient. An example in this respect was the Estonian experience of cross-border collaborations sharing best practices and establishing uniform standards for mass digitisation (see box 4). Without the Recommendation, it is less likely that this cross-border collaboration would have taken place to this extent.

As highlighted in the evaluation study, the promotion of coordinated approaches on legal deposit arrangements is a part of the scope effect of the Recommendation. This led to increased efforts enabling CHIs to access a wide variety of materials.

With respect to Europeana, the Europeana evaluation report highlights an overall appreciation of the European added-value of the Europeana initiative by cultural heritage institutions across the EU, bringing them together and enabling them to collaborate and share their material. This has not only led to the sharing of best practices on common standards, but also helped nurture a European network of data partners, aggregators and professionals in various fields which has stimulated capacity building and the exchange of expertise.

In addition, the findings of the workshop of January 2020 point to the need to maximise the impact of Europeana as an enabler of digital transformation in the cultural heritage sector. This could be the case for instance by using the multiplier effect, developing national strategies or making public funding conditional on the adoption of standards. This would give the Recommendation more added value.

Also, the findings of the DCHE workshop of November 2019, together with the targeted consultation of the representatives of the Member States in the DCHE, confirmed that the contribution of the Recommendation in strengthening the role of Europeana and more generally in the increase of digitisation, online access and preservation of digitised cultural material. It is important to note that 87% of the respondents to the online public consultation considered the EU’s support to Europeana as (very) important.

The reporting system laid down in the Recommendation has been an important means to create a structured process for its implementation (e.g. to incentivise the focus on priorities and to collect information), in addition to giving CHIs and authorities the opportunity to provide feedback.

Finally, according to the majority of the respondents 153 (98%) to the online public consultation, the EU and Member States should do more to help cultural heritage institutions address the challenges and seize the opportunities of the digital era. A large majority (around 49% strongly agree and 32% agree) considered that the digital transformation of European cultural heritage institutions should be addressed and coordinated at the EU level.

6.Conclusions

This section provides the conclusions of the evaluation, highlighting what is working and not working with the EU intervention and why, and what would be the most appropriate future course of action concerning the Recommendation, i.e. whether the identified issues need to be addressed by action at EU level or can be resolved by action carried out by Member States/CHIs over time.

While its implementation by Member States varies to a large extent, the Recommendation led to an overall improvement in the field, as acknowledged by the various reporting and consultation activities. This stems, for instance, from i) the adoption of national strategies and/or funding programmes for the digitisation of cultural heritage assets; ii) the increased number of PPPs; iii) greater visibility and closer cooperation among CHIs; iv) practical measures to optimise the use of digitisation capacity; v) increased volume of the digitised cultural heritage resources and the positive overall impact of Europeana. The Recommendation’s objectives have been reached and remain relevant to a large extent. Several factors contributed to this, such as tools designed to support CHIs or technological advancements that promote open-source software and shared data, while other factors hindered the achievement of the objectives (e.g. lack of adoption of common standards and approaches, insufficient advanced digital skills and literacy).

Despite insufficient data being available to quantify the costs triggered by the Recommendation’s implementation and to make a comparison across the Member States, stakeholders were positive about the intervention’s efficiency, considering that overall the benefits outweigh the costs.

In terms of relevance, there is a general acknowledgement that the Recommendation was useful in responding to the sector’s needs, as identified at the time of its adoption.

Likewise, the Recommendation has proved to be coherent not only with other initiatives at the EU level, such as the initiatives on copyright or more generally on culture and cultural heritage, but also with the actions and objectives that Member States promoted at national level.

The Recommendation’s added value (e.g. of increasing digitisation and online access to digital resources, of encouraging closer cooperation and promoting innovation and the transfer of ideas across the EU) was also highlighted during the consultation activities.

Nonetheless, in order to increase the intervention’s effectiveness, efficiency, relevance, coherence and added value, the evaluation process has pointed to a number of important aspects for the future course of action.

The cultural heritage landscape has changed significantly over the past years, bringing new needs, but also new opportunities. In particular, new political, technological and socio-economic parameters have brought about new challenges. Moreover, in a general context where the entire world is struggling to overcome an unprecedented crisis and to define robust policies to recover and rebuild our economies, our society turns more than ever to the cultural heritage sector for its rich resources for a variety of purposes. A clear message that emerged from the consultation activities was that while the sector needs to turn the crisis into an opportunity for more growth and social cohesion, it must also be equipped with the right tools to do so. More particularly, CHIs need to be powered by digital technologies, embrace the digital transformation and thus lead the change. 

The COVID-19 crisis has underlined once more the need for and importance of improving online access to digitised cultural material and improving its use/reuse; this is true for the CHIs, for the creative sectors and for society at large. The crisis has been a catalyst for more online activities and services, but has also led to some inequalities within the sector and thus more missed digital opportunities. Funding, a persistent challenge for the cultural heritage sector before the crisis, has become more stringent in the current context for many CHIs and even more vital to their survival.

Advanced or emerging digital technologies can empower the sector to unlock the potential of the cultural heritage sector through a more widespread adoption of 3D, AR/VR, AI, data, etc., while providing new ways for cultural objects to be shared and used. They allow visitors to visit different and new places regardless of their location and cost, they offer a more appealing and engaging experience. They support our cultural heritage institutions in narrating European stories 154 .

Higher quality of the digitised cultural heritage resources would enhance innovation and creation through the use and reuse of digitised cultural material across the sectors. The current focus mostly on increasing the quantity of the digitised assets has to go together with higher quality.

Further adoption of common standards and approaches for digitised content would remove some of the remaining interoperability related obstacles, while facilitating cooperation and exchanges that strengthen the capacity for innovation. Europeana has been key in strengthening cooperation and standardisation activities across borders with the overarching goal of increasing the use of standards in digitising and sharing digitised cultural material throughout Europe. The initiative’s involvement has also been vital for promoting open cultural data and it should continue to push forward standardisation activities to facilitate the work of the cultural heritage institutions.

At the same time, Europeana has to play a more prominent role in enabling CHIs to seize the opportunities provided by the digital transformation in the cultural heritage sector. Europeana’s three strategic priorities are to strengthen its infrastructure, improve the quality of data and content and support the cultural heritage sector to engage in digital transformation. The results of the latest consultation activities have confirmed these and particularly the role of the initiative in supporting cultural heritage institutions to embrace digital.

Broadening the scope of the Recommendation to cover key cultural heritage marginally addressed or not addressed at all yet, like immovable, born digital and intangible, would create a more complete framework of action for the CHIs and would be in line with the feedback from the online public consultation.

Advanced digital literacy and skills have become paramount for CHIs, in a context where the COVID-19 pandemic speeded up the importance of finding new ways of working and connecting to audiences and sharing creative content online. It has also been a recurrent issue raised in the consultation activities.

Making full use of cultural heritage data for the benefit of the sector by for instance promoting national strategies and cross-sectoral collaboration that encourage the creation of high value datasets would be beneficial not only to the cultural sector, but to other sectors as well (e.g. research, sustainable tourism, education) and would contribute to building the data space for cultural heritage.

European efforts to further digitise and broaden online access of cultural heritage have been recognised at international level, be it through the acknowledgement of the Member States Declaration of 2019 by UNESCO or through the work of the Europeana initiative with international organisations, such as the World Digital Library or the Digital Library of America. Further partnerships in the field would help in finding shared responses to common challenges and exchanging best practices as well as showcasing and promoting European culture, values and success stories. Europeana is, as mentioned above, a unique initiative offering material in many languages. It has a lot to offer in terms of experience and knowledge.

Some of the aspects above may be further addressed by the Member States alone, although it is not certain to what extent, especially in the light of the wide-ranging impacts of the COVID-19 crisis and thus of the declining budgets assigned to cultural heritage-related activities. However, the evaluation analysis (including the outcome of the consultation activities) points to a strong impact of an EU intervention supporting the efforts of the Member States.

Annex 1: Procedural information

1.Lead DG, Decide Planning/CWP references

Directorate General Communication Networks, Content and Technology (CNECT) is the lead DG for this evaluation, and in particular Unit G2 – Interactive Technologies, Digital for Culture and Education.

The DECIDE reference is: PLAN/2019/5584.

2.Organisation and timing

In accordance with the Better Regulation Guidelines, an ISG was set up with representatives from various Directorates General and services of the Commission 155 .

The meetings of the steering group were chaired by DG Communications Network, Content and Technology. The first meeting took place on 17 September 2019 and laid the ground for the work related to the evaluation. The ISG was regularly consulted over the course of the evaluation, typically in conjunction with the submission of specific draft reports by the contractor responsible for carrying out the external study. The ISG was also consulted during the drafting of this staff working document, which includes the various remarks made.

The evaluation was extended, given the fact that the public consultation was launched later than initially planned, due mostly to the COVID-19 crisis (the public consultation was online from 22.06.2020 to 14.09.2020).

3.Exceptions to the better regulation guidelines

No exceptions from the usual procedure laid down in the Better Regulation Guidelines requirements were requested/made.

4.Consultation of the RSB:

Not applicable.

5.Evidence, sources and quality

Most of the evidence was collected with the support of an external contractor (see “Study to support the evaluation and possible revision of the Commission Recommendation (2011/711/EU) on the digitisation and online accessibility of cultural material and digital preservation, conducted by ICF, Final Report, March 2020).

The evidence is based on external expertise and knowledge from Member States’ competent authorities, cultural heritage institutions, international and umbrella organisations, the Europeana ecosystem of organisations and professionals, technology stakeholders with research, products and services in ICT for cultural heritage, citizens, through the numerous consultation activities held by the Commission (see Annex 2).

However, the quantification of the costs and benefits of the intervention was difficult, due to the diffuse and intangible nature of the benefits of cultural heritage, and to the lack of available or updated data on costs in Member States (see Section 4.3). Alternative proxy data were used and explicitly mentioned in the analysis where quantitative data are missing.

The overall evaluation is considered robust and thorough.

Annex 2: Synopsis report of the consultation activities

1.INTRODUCTION

The Commission Recommendation on the digitisation and online accessibility of cultural material and digital preservation (2011/711/EU) 156 (“the Recommendation”) is the Commission’s main policy tool in the area of digital for cultural heritage, aiming to optimise the use of information and communication technologies to unlock the full economic and cultural potential of Europe’s cultural heritage.

The Recommendation dates back to October 2011. Some of the challenges facing the cultural heritage sector at that time (e.g. an urgent need to protect and preserve European cultural heritage in danger), are still present today, but there are also others that have arisen since (e.g. major advances in digital technologies). Moreover, the COVID-19 pandemic confirmed once more the importance of enabling the cultural heritage sector to seize the opportunities provided by digital technologies.

It was therefore necessary to assess the Recommendation and decide whether it still met the needs and expectations of the cultural heritage sector, of the users, creative sectors, citizens and society in general. A clear, evidence-based assessment of the Recommendation took place in order to support the Commission’s decisions on the next steps, shaping an appropriate policy framework to preserve Europe’s valuable cultural assets.

To this end, a wide stakeholders’ consultation on digitisation and digital transformation in the cultural heritage sector was necessary.

The following consultation activities were organised:

1.Feedback on the evaluation roadmap

2.Several workshops

3.Targeted consultation of the Expert Group on Digital Cultural Heritage and Europeana (DCHE) representatives

4.Delphi panel

5.Open Public Consultation

The feedback received from the consultation activities, in particular on the effectiveness, efficiency, relevance, coherence, EU-added value of the Recommendation, has been integrated in the evaluation, while various key issues raised by stakeholders will be considered for the future policy framework (i.e. for the revision of the Recommendation).

2.CONSULTATION PROCESS AND ANALYSIS OF RESPONSES

2.1. Feedback on the evaluation roadmap

An evaluation roadmap was published for a four-week feedback period (from 29 July 2019 to 26 August 2019). It aimed to inform citizens and stakeholders about the Commission’s work, encouraging them to provide feedback and to participate effectively in consultation activities. The evaluation roadmap and the feedback from stakeholders can be consulted on the “ Have your Say” page of the initiative .

The evaluation roadmap aimed to inform citizens and stakeholders about the Commission’s work, encouraging them to provide feedback and to participate effectively in consultation activities.

A total of eight responses were received from a variety of stakeholders (e.g. business organisations, non-governmental organisations, public authorities, citizens).

Respondents acknowledged the positive impact of the Recommendation so far, including the significant changes and benefits for the cultural heritage sector. Stakeholders welcomed the evaluation of the Recommendation in the current context, defined by a rapid progress of advanced digital technologies.

Several stakeholders also mentioned bottlenecks and issues to be addressed in the future, such as quality of metadata, infrastructure to disseminate digitised material, copyright and licensing issues.

2.2. Workshops

a) Two workshops were organised in the context of the 6th Meeting of the Expert Group on Digital Cultural Heritage and Europeana (5-6 November 2019, Luxembourg). The first one specifically focused on both backward (impact/functioning of the Recommendation) and forward-looking aspects (i.e. areas for improvement).

According to the participants, the Recommendation has contributed to raising awareness about the cultural heritage sector, to facilitating more specific policies, strategies and projects at national and local levels. The Recommendation has contributed to more cooperation, openness and sharing, to more interoperability, while strengthening the development of Europeana and its activities.

The respondents also underlined the need to address specific areas, such as better quality of data and metadata, enhanced interoperability, more effective digital preservation policies, and enhanced capacity building. Moreover, the Recommendation should have a more holistic approach and scope and cover other categories of cultural heritage, such as born-digital cultural heritage. The importance of Europeana has been highlighted, as well as the need to maximise its role as enabler of digital transformation.

The second workshop , more focused on the Declaration of Cooperation on advancing the digitisation of cultural heritage 157 , called for views on 3D documentation, 3D repository, technology take-up and capacity building and the instruments contributing to those. Respondents considered building on existing standards and/or processes important, while the elaboration of common standards should incorporate the fundamental principles of usability and openness. The need for advanced digital skills and literacy has also been highlighted, along with the need for increased knowledge sharing. Participants pointed to the need for European and national funding to encourage and stimulate the adoption of advanced digital technologies. Cooperation at international level has been considered important, both from a content and a technology point of view.

b) Workshop in the context of the Europeana Conference (28 November 2019, Lisbon)

The workshop took place during the Europeana Conference in Lisbon on 28 November 2019 and explored similar topics to the first two workshops, but delving more in-depth based on the responses and gaps identified in the first workshops.

Participants pointed to the fact that the low levels of digital literacy and skills remain a challenge, particularly for galleries, libraries, archives, and museums (GLAMs), while retaining expert staff is still difficult for cultural heritage institutions. The sector needs more capacity building and Europeana should have an enhanced role in this, by sharing good practices and expertise. Cultural heritage institutions have been struggling to digitise and preserve particularly intangible cultural heritage. Participants mentioned the challenge of ensuring better interoperability in the sector (for instance between various formats used and Europeana Data Model). Cultural heritage institutions need more robust and sustainable funding, the smaller ones in particular. Quality of digitisation should also become a priority, contrarily to the current approach which focused mainly on quantity. It was emphasised that digital transformation requires coordinated efforts, developing holistic approaches at all levels and a stronger collaboration between institutions. Europeana should have an enhanced role in the context of digital transformation, including by promoting further standardisation, a multilingual access, exploring crowdsourcing opportunities and securing alternate funding streams, while building bridges in the sector.

c) Workshop with the international organisations on digital for cultural heritage (10 December 2019, Brussels)

Twelve representatives of various international organisations and networks took part in a workshop on the orientations and priorities for EU funding, activities and policy in the area of digital for cultural heritage (including on the Recommendation).

Participants raised the need for enhanced capacity building, to ensure better training and knowledge of staff (including to enable them to run digitisation projects and evaluate what can be reused), for increased cross-border digitisation efforts. Further development of common standards (relating to 3D, metadata enrichment, multilingual access) was also mentioned. The use of advanced technologies would maximise the impact needed in the sector. The quality of content should be an important part of any policy framework.

The workshop emphasised that the Recommendation would be more effective if it strengthened the networks of the cultural heritage institutions more, enhanced the knowledge exchange and best practice, including by an increased collaboration at international level (e.g. by providing more support for transfer of technology, development of skills in third countries).

d) Workshop with stakeholders on digital transformation in the cultural heritage sector (10 January 2020, Brussels)

Various stakeholders (cultural heritage institutions, international organisations in the field of culture, research organisations, etc.) took part in this workshop, building on the information collected during previous workshops.

According to the feedback provided by stakeholders, more needs to be done to advance digital literacy and skills in the cultural heritage sector. The Recommendation does not sufficiently address the need to strike a balance between quality and quantity of the digitised material, while the sector needs to move forward from an approach focused mostly on quantity.

Stakeholders highlighted that the current Recommendation does not sufficiently address the need to make full use of cultural heritage data for the benefit of the cultural heritage sector and other areas. There is room for improvement to increase the collection, access, use and reuse of cultural heritage data. Data sharing among cultural heritage organisations should be further enhanced.

With regard to Europeana, stakeholders underlined its positive impact on the cultural heritage institutions. Further partnerships and cooperation at international level should help to find shared responses to common challenges and exchange best practices, while promoting European culture, values and success stories.

2.3. Targeted consultation of the Expert Group on Digital Cultural Heritage and Europeana representatives

A targeted consultation of the Expert Group representatives was organised between November 2019 and January 2020, to complement previous consultations and to build a coherent and cohesive view of both the Recommendation’s implementation and possible options for the future.

A total of 20 respondents, from 18 Member States, completed the targeted consultation.

The targeted consultation suggested that the Recommendation had been overall effective, for instance in helping to raise additional funds for digitisation. Respondents agreed that the Recommendation had been relevant for the needs identified originally, while also widely acknowledging the changes in the context in which cultural heritage institutions carry out their activities now. This is mainly due to technological advancements, which need to be better reflected in a future revision of the Recommendation.

The Expert Group representatives emphasised the importance of increasing the quality of digitised cultural material, of providing common (comparable, interoperable) standards, strengthening the cooperation among cultural heritage institutions and with private sector, having wider online access to digitised cultural material, for use and reuse, reinforcing national strategies for long-term preservation of digital material, updating action plans implementing the strategies.

2.4. Delphi panel

An online Delphi panel composed of experts from the Europeana ecosystem, cultural heritage institutions, research/academia, Member States was launched to support and validate the identification of challenges/problems in the sector, and the most appropriate ways to respond to those.

The panel considered several scenarios for the next steps, such as the situation where no policy change would take place; updating the specific provisions in the Recommendation; updating specific provisions and expanding the scope of cultural material covered by the Recommendation; updating provisions, expanding the scope and addressing the digital transformation of cultural heritage institutions. The outcome of this panel was that a more complex policy initiative that would update the relevant provisions, expand the scope covered so far and address digital transformation would have higher (positive) impact, including in terms of efficiency and effectiveness.

2.5. Open Public Consultation

The public consultation opened 22 June 2020 and closed on 14 September 2020. The aim of the consultation was to gather stakeholders’ opinions on digitisation and digital transformation in the cultural heritage sector, as well as on the Commission’s Recommendation on the digitisation and online accessibility of cultural material and digital preservation (2011/711/EU). The consultation collected views from citizens, public administrations, cultural heritage institutions, international organisations and networks, the Europeana ecosystem, companies, research organisations, and academia. 

The questions covered in the consultation focused on: 

a)the importance of digitisation and digital transformation in the cultural heritage sector and the ways of supporting such processes.

b)the relevance and future of the Recommendation.

Who participated to the consultation?

There were 565 respondents, including 186 citizens, 145 academic/research institutions, 68 public authorities, 53 companies/business organisations, 37 NGOs, and 75 other interested parties.

Error! Reference source not found. Respondents by stakeholder group.

Source: Open public consultation on digital access to European cultural heritage. N=564

About half of the respondents indicated that they are active in the culture sector (261 responses), followed by education or research (169), Technology/ICT (60), Creative arts/media (29), Tourism (3). 42 respondents indicated that they are active in another sector.

Figure 2. Q2. In which sector are you or your organisation active?

Source: Open public consultation on digital access to European cultural heritage. N=564

Cultural heritage sector in the context of the COVID-19 crisis

The crisis affected organisations mostly through loss of income. While the majority did not have to dismiss staff, many of them had to suspend contracts with freelance workers.

The tourism sector appears to have been most severely hit by the crisis, with almost two thirds of respondents referring to dismissal of staff. On the other hand, results by types of stakeholders show that the public sector was the least affected by the crisis in relation to the staff dismissed.

Almost half of the respondents noted that they struggled with advanced digital skills needed to manage the online presence, particularly in the education/research and cultural sectors. The pandemic had a significant impact on the digital activities of the respondents. Many of them changed their online offer and made more use of digital technologies. This resulted in an extended online access to curated audio-visual assets, virtual tours of exhibitions, more academic research material (e.g. journals, books), increased social media activities, online collaboration tools (e.g. for workshops, conferences) and e-learning materials. According to the respondents, the majority of these online offers/services will remain in place after the pandemic and will most likely be expanded.

Based on lessons learned from the COVID-19 crisis, most of respondents (98%) considered digital technologies as (very) important for making their organisation more resilient in the future (as shown in figure 3).

Figure 3. Question 7. Based on the lessons learned from the COVID-19 crisis, how important do you think digital technologies will be in order to make your organisation more resilient in the future?

Source: Open public consultation on digital access to European cultural heritage N=557

Moreover, the majority are already making use of digital technologies to overcome the difficulties posed by the COVID-19 related measures. However, the respondents pointed out that the use of these technologies mostly depends, in many cases, on the funding available.

Digitising cultural heritage

There was a strong agreement on the importance of digital technologies for cultural heritage. Overall, a majority of respondents strongly agreed (67%) or agreed (27%) that digital technologies can help in the reconstruction of damaged cultural heritage, 97% agreed that digital technologies can help to increase the number of people accessing cultural heritage through the Internet, while for 98% digitisation, online access and digital preservation of cultural heritage are important to society. There were no major differences identified by types of stakeholders.

A large majority of respondents considered 3D technologies valuable for creating digital twins of cultural heritage of buildings, monuments and sites (92%), for creating high-quality 3D models of museum objects (91%), for creating 3D models of cultural heritage buildings, monuments and sites or museum object for online visualisation (93%) or immersive experiences (88%).

According to certain respondents, additional areas where the application of 3D technologies would be valuable are education sector (to facilitate collaborative research e.g. virtual archaeology) and creative industry (e.g. for films, video games and other virtual reality projects).

Using and reusing digitised cultural heritage assets

Respondents were asked to express their agreement to a set of statements related to using and reusing digitised cultural heritage assets, based in particular on the lessons learned from the COVID-19 crisis.

Improving access to digital cultural heritage was perceived as very important overall and across all sectors and stakeholder types (see figure 4). Findings show that the availability of digitised cultural heritage content for reuse is mostly perceived very important in the research and education sectors.

Figure 4. Question 14. In your opinion, in the light of the COVID-19 crisis, how important is it to improve online access to digital cultural heritage?

Source: Open public consultation on digital access to European cultural heritage N=559

In terms of ways to present cultural heritage, the most appealing to respondents were curated content (e.g. online exhibitions), tutorials/online classes on specific topics and virtual visits of sites (e.g. 3D or VR).

Respondents highlighted the main difficulties and shortcomings in relation to accessing and reusing European heritage content online, including insufficient quality (e.g. low-resolution images, or poor or no explanatory text), unclear copyright and re-use status of digital objects, insufficient content available.

The vast majority of respondents (90%) considers that digital technologies empower and encourage people into more active participation such as developing new content or uploading their cultural content online, or blogging about it.

Digital transformation

Digital transformation has been the issue that most of the respondents considered important in the context of a future revision of the Recommendation.

In the light of the COVID-19 crisis, providing online access to digitised content (81%) and creating a more agile digital environment (71%) were considered to be the most important aspects of digital transformation in the cultural heritage sector.

With regard to digital skills in the aftermath of COVID-19, data and metadata management and analytics, copyright and licensing, followed by digitisation are the core skills set most of respondents consider important to acquire.

On the applications of data and AI technologies in the area of cultural heritage, the vast majority of respondents (91%) found the automated annotation of digitised cultural heritage to be (very) valuable in the area of cultural heritage (in particular by respondents from the tourism and creative arts/media sectors). Around 90% of the respondents found the automated recognition of objects and fragments of objects to be (very) valuable, followed by the extraction of new knowledge from cultural heritage data (89%) and the reconstruction of lost cultural heritage information (85%).

On the digital infrastructure elements, 96% of respondents consider common data spaces for sharing cultural heritage data and building high-value data sets to be (very) valuable. 97% of respondents considered repository infrastructure for digitised cultural heritage content to be (very) valuable, followed by cloud infrastructure for research on digitised cultural heritage material (90%) and supercomputing capabilities for large-scale cultural heritage simulations (76%).

The majority of respondents (98%) consider that the EU and Member States should intensify their actions in order to help cultural heritage institutions address the challenges and seize the opportunities of the digital era and (81%) agreed that the digital transformation of European cultural heritage institutions should be addressed and coordinated at the EU level

The EU’s support to Europeana, Europe’s only digital platform for cultural heritage was perceived important by 87 % of the respondents.

Specific questions about Recommendation 2011/711/EU

·Effectiveness: 41% of respondents either agreed (33%) or strongly agreed (8%) that the provisions of the Recommendation have been implemented effectively and have achieved their objective to improve conditions in the areas addressed.

·Efficiency: 70% of respondents either agreed (44%) or strongly agreed (26%) that the benefits of implementing the Recommendation are significant and justify the costs, which are proportionate. with this statement

·Coherence: 59% of respondents either agreed (45%) or strongly agreed (14%) that the Recommendation has been coherent with other EU policies and initiatives. Approximately with this statement.

·Relevance: 95% of respondents either strongly agreed (70%) or agreed (25%) that the Recommendation should be updated to reflect today’s needs in the area of digitised cultural heritage and increase the potential of Europe’s cultural heritage.

·EU added-value: 80% of respondents either agreed (45%) or strongly agreed (35%) that the Recommendation provides added value compared to initiatives that Member States would have taken in the absence of it.

Annex 3: Methods and analytical models

This annex provides additional details on the main methodological steps of the evaluation, i.e. inception phase, data collection, data analysis, synthesis and conclusions.

1) Inception phase:

This phase laid down the foundations for the subsequent stages of the evaluation. It included the preparation of the intervention logic and of the evaluation questions on which the evaluation was built. The evaluation questions are presented at the end of this annex.

This phase included also the meeting with the contractor and the ISG members to lay down the basis for the process and deliverables.

2) Data collection and 3) Data analysis:

An evaluation roadmap  defining the scope of the evaluation was published between 29 July 2019 and 26 August 2019 for feedback. The Commission received eight contributions from the Swedish National Heritage Board (Sweden), Europeana Foundation (Netherlands), Land Salzburg koordiniert vom Landes-Europabüro (Austria), Federation of European Publishers (Belgium), Wikimedia (Bulgaria), Jukka S. RANNILA (Finland), Andre REBENTISCH (Germany) and a citizen. The Commission took these comments into account to the extent possible and informed the evaluation.

The evaluation study, i.e. Study to support the evaluation and possible revision of the Commission Recommendation (2011/711/EU) on the digitisation and online accessibility of cultural material and digital preservation was conducted by ICF between August 2019 and March 2020 and updated at a later stage, in November 2020, following the online public consultation. The general objective was to carry out an evidence-based assessment of the results and impacts of Recommendation across the EU and to assess the potential impacts of a possible revision of the Recommendation. Data weregathered through desk research and stakeholder consultation activities. More information on data collection in Annex 2 of the evaluation study.

A consultation strategy has been created and agreed with the ISG. The consultation activities aim at collecting the views of Member States’ competent authorities, cultural heritage institutions, international and umbrella organisations, the Europeana ecosystem of organisations and professionals, technology stakeholders with research, products and services in ICT for cultural heritage, and citizens. All these different stakeholder groups are expected to have important information and insights on actions taken for the Recommendation’s implementation and its results, as well as an interest in and opinions on shaping the debate about the possible options for the future.

The consultation strategy has two main objectives: (1) collect views on the Recommendation’s implementation(to support the analysis on the retrospective evaluation of the Recommendation) and (2) collect views on the impacts of possible future changes to the Recommendation (to support the forward-looking assessment).

The Commission held a series of workshops with representatives of Member States, stakeholders, international organisations and experts in order to collect relevant and solid information on the impact of the Recommendation, as well as on the challenges and needs of the sector in the context of the digital transformation, be addressed in the future. More particularly, the Commission held:

a)An workshop on the future of the Commission Recommendation of 27 October 2011 on the digitisation and online accessibility of cultural material and digital preservation (2011/711/EU), in the context of the DCHE group meeting, November 2019, Luxembourg;

b)An workshop on the implementation of the Declaration of Cooperation on advancing the digitisation of cultural heritage, in the context of DCHE group meeting (November 2019, Luxembourg);

c)An workshop on the digital transformation in the cultural heritage sector in the context of the Europeana conference ( November 2019, Lisbon) 158 ;

d)An workshop on the digital cultural heritage, with the internationals organisations and networks relevant for the sector (December 2019, Brussels);

e)An workshop on Digital transformation in cultural heritage 159 , (January 2020, Brussels) 160 .

Between November 2019 and January 2020, the Commission held a targeted consultation of the DCHE representatives, aiming to collect data from stakeholders with the most expertise on the Recommendation (and where applicable national cultural heritage institutions, depending on each Member State), and including questions covering all five evaluation criteria. The Commission received 20 responses, from 18 Member States, that informed the evaluation study.

Between 22 June and 14 September 2020, the Commission held an open public consultation,  focusing on important issues for the sector, such as the importance of digitisation and digital transformation in the field and the ways of supporting such processes, as well as on the impact of the Recommendation. The objective was to collect diverse opinions and experiences from all stakeholder groups. The consultation was available in all the EU official languages 161 and gathered 565 responses overall. The results of the online consultation are analysed in a separate Annex (see Annex X) to this report) as well as integrated into the evidence base for the evaluation. No significant limitations have been encountered.

An additional main source of information for the study and the evaluation were the national implementation reports provided by the Member States and assessed and consolidated by the Commission in a (consolidated) progress report on the Recommendation’s implementation (see Section 2).

4)    Synthesis and conclusions

The final steps of this phase mainly consist of:

·an assessment of the outcomes of the evaluation study, of the consultation activities and of the implementation reports, followed by

·conclusions on the impact of the Recommendation in the cultural heritage sector across the EU and more particularly on its effectiveness, efficiency, coherence, relevance and EU added-value.

Annex 4: Evaluation criteria and questions

Effectiveness criterion

Efficiency criterion

Relevance criterion

Coherence criterion

EU-added value criterion

(1)

  https://ec.europa.eu/growth/sectors/tourism/offer/cultural_en

(2)

See report by Network of European Museums Organisation (NEMO)

(3)

According to the report prepared by NEMO (see footnote 2), 4 out of 5 museums have increased their digital services to reach their audiences.

(4)

See also the Art Fund survey on COVID-19 Impact: Museum sector research report on the challenges facing museums and galleries, which highlights that the vast majority (86%) of organisations have increased their online presence.

(5)

  OJ L 283, 29.10.2011, p. 39–45

(6)

  https://ec.europa.eu/info/law/law-making-process/planning-and-proposing-law/better-regulation-why-and-how/better-regulation-guidelines-and-toolbox_en

(7)

Study to support the evaluation and possible revision of the Commission Recommendation (2011/711/EU) on the digitisation and online accessibility of cultural material and digital preservation,

SMART 2019/0027.

(8)

Communication from the Commission “Europe 2020 -A strategy for smart, sustainable and inclusive growth”, COM(2010) 2020 final, 3.3.2010.

(9)

OJ L 236, 31.8.2006, p. 28–30, OJ L 118M , 8.5.2007, p. 1279

(10)

  See “The new renaissance -Report of the Comité des Sages on bringing Europe's cultural heritage online ”, E. Niggemann, J. De Decker, M.Lévy, 2011

(11)

European Commission (2010). Second progress report on the digitisation and online accessibility of cultural material and on digital preservation in the European Union

(12)

Based on ENUMERATE. Digitisation in Cultural Heritage Institutions, Survey Report (2012).  

(13)

According to the evaluation study 23 % of institutions had a written digital preservation strategy, while around one-third were included in a national preservation strategy

(14)

See Commission’s second progress report (2010)

(15)

See also the Council conclusions of 10 May 2012 on the digitisation and online accessibility of cultural material and digital preservation (2012/C 169/02), OJ C 169, 15.6.2012, p. 1–4

(16)

European Commission (2018), Cultural heritage: digitisation, online accessibility and digital preservation.  Consolidated progress report on the implementation of Commission Recommendation, 2015-2017

(17)

  ENUMERATE 2017  

(18)

  Final report on Digitisation and IPR in European Museums , published by the Network of European Museum Organisations in July 2020. The survey included 3 different target groups; national museums organisations, national ministries in charge of museums and museums, in Austria, Belgium, Croatia, Estonia, Finland, France, Germany, Greece, Italy, the Netherlands, Poland, Slovenia and Spain. In total 60 museums in Europe, 13 Member States, plus Serbia and the UK participated in the survey.

(19)

Art and design museums have an average of 65% of their collections digitised, natural history museums 15%, history and archaeology museums 27%. One of the reasons for such differences could be the existence of complex 3D objects that need the appropriate technology and financial resources.

(20)

  Directive 2012/28/EU of the European Parliament and of the Council of 25 October 2012 on certain permitted uses of orphan works, OJ L 299, 27.10.2012, p. 5–12

(21)

Europeana Foundation (2013). Europeana celebrates 5 years and 30 million objects. Available at: https://pro.europeana.eu/post/europeana-celebrates-5-years-and-30-million-objects .

(22)

Sound: 805,897 and video: 446,092.

(23)

Quality standards and how the content can be used are outlined in the Europeana Publishing Framework ( https://pro.europeana.eu/post/publishing-framework ). The framework has four tiers of criteria on the quality of the content. Three tiers of criteria on the quality of metadata were added in October 2019.

(24)

  Report on the evaluation of Europeana and the way forward , 6 September 2018, COM(2018) 612 final

(25)

  Directive (EU) 2019/790 of the European Parliament and of the Council of 17 April 2019 on copyright and related rights in the Digital Single Market and amending Directives 96/9/EC and 2001/29/EC , OJ L 130, 17.5.2019, p. 92–125

(26)

Moreover, 17% of respondents were not in agreement with the statement, while 35% of respondents neither agreed nor disagreed with this statement, 8% did not know.

(27)

The extent to which the objectives have been achieved and the achievements attributed to the Recommendation is assessed across the whole section.

(28)

European Commission (2014). Cultural heritage: digitisation, online accessibility and digital preservation. Consolidated progress report on the implementation of Commission Recommendation, 2011-2013.

(29)

Commission’s consolidated progress report (2018)

(30)

 European Commission (2019) Workshop Report - DCHE workshop on the future of the Commission Recommendation of 27 October 2011 on the digitisation and online accessibility of cultural material and digital preservation (2011/711/EU), 5 November 2019 and targeted consultation

(31)

Workshop Brussels (January 2020), workshop Lisbon (November 2019)

(32)

Forbes, N. and Fresa, A. (2016). Public-Private Partnerships for cultural heritage: opportunities, challenges, future steps, Think Papers Collection 07. Available at: https://www.digitalmeetsculture.net/wp-content/uploads/2016/04/rch_thinkpapers_07.pdf

(33)

MS58 Study on the costs and funding of digitisation, M24 (Europeana DSI-4) by Europeana Foundation, August 2020

(34)

Brussels, January 2020

(35)

See consolidated report 2018. Moreover, the workshop held in Lisbon in 2019 emphasised the opportunities offered by crowdsourcing and alternative funding streams.

(36)

See consolidated report 2018

(37)

Evaluation study incorporating information from the targeted consultation

(38)

Workshop Lisbon (November 2019)

(39)

See also evaluation study

(40)

See the evaluation study

(41)

Report workshop DCHE, November 2019

(42)

Consultation activities

(43)

Evaluation study, international GLAM Labs Community. Available at: https://glamlabs.io/books/open-a-glam-lab/

(44)

Workshop Brussels (January 2020)

(45)

Workshop Lisbon (November 2019)

(46)

Workshop Brussels (January 2020)

(47)

Article 6 of this Directive requires Member States to introduce an exception to copyright and related rights in their national laws in order to allow CHIs to make copies of works and other subject matter for preservation purposes. Articles 8 to 11 of the Directive provide a legal solution for the digitisation and dissemination of works that are considered to be out of commerce. Article 14 of the Directive ensures that reproductions of public domain works of visual art cannot be protected by exclusive rights unless the reproduction material is itself original. This Article prevents CHIs from claiming copyright protection on digital copies of works of visual arts that are no longer protected under EU copyright law.

(48)

Feedback from the online public consultation

(49)

Workshop Brussels, January 2020

(50)

  https://pro.europeana.eu/project/3d-content-in-europeana

(51)

See the evaluation study

(52)

Consolidated report 2018

(53)

  https://www.europeanschoolnetacademy.eu/courses/course-v1:Europeana+Culture_EN+2020/about

See also the report “ Five years promoting innovative learning with digital cultural heritage ”, September 2020  

(54)

CC0 is used to waive all the rights in a digital object. By applying this waiver, all possible existing rights in the content are waived, and the objects can be used by anyone without any restrictions. Europeana pro provides additional details on this.

(55)

See Report on the evaluation of Europeana.

(56)

Targeted consultation DCHE

(57)

Evaluation study

(58)

Consolidated report 2018; European Commission (2016). Cultural heritage: digitisation, online accessibility and digital preservation . Consolidated Progress Report on the implementation of Commission Recommendation (2011/711/EU) - 2013-2015 .

(59)

Evaluation study

(60)

Evaluation study

(61)

Evaluation study

(62)

Europeana Impact Playbook: https://pro.europeana.eu/resources/standardization-tools/impac

(63)

  https://weareculture24.org.uk/

(64)

The GIFT BOX was a result of the Gift Box project, funded by the EU, for more details see: https://gifting.digital/

(65)

http://pro.europeana.eu/publication/publishing-framework

(66)

Workshop Brussels (January 2020)

(67)

A more in-depth analysis will be made in Section 5.

(68)

See evaluation study

(69)

Workshop Lisbon (November 2019).

(70)

Workshop Brussels (January 2020). 

(71)

MS58 Study on the costs and funding of digitisation, M24 (Europeana DSI-4) by Europeana Foundation, August 2020

(72)

See evaluation study

(73)

'N' indicates national and 'EU' indicates largely (but not exclusively) European funding.

(74)

Flanders.

(75)

French community.

(76)

DFG (German Research Foundation).

(77)

DFG (German Research Foundation).

(78)

Baden-Württemberg state.

(79)

Berlin municipality.

(80)

Brandenburg state.

(81)

Saxony state.

(82)

SGCB (General Sub-Directorate for Library Coordination).

(83)

SGCB (General Sub-Directorate for Library Coordination).

(84)

Ministry of Defence.

(85)

Ministry of Defence.

(86)

Autonomous community – Galicia.

(87)

Spanish State Archives.

(88)

Spanish Cultural Heritage Institute.

(89)

Province of Friesland.

(90)

The ENUMERATE surveys capture costs incurred annually by CHIs for creating, acquiring, maintaining, enhancing and preserving the digital collections and are available at: https://pro.europeana.eu/page/past-surveys  

(91)

This is an estimate of all costs related to the initial creation, ongoing maintenance, enhancement and preservation of the digital collections.

(92)

Evaluation study

(93)

MS58 Study on the costs and funding of digitisation, M24 (Europeana DSI-4) by Europeana Foundation, August 2020

(94)

Report DCHE workshop November 2019 and targeted consultation

(95)

  https://pro.europeana.eu/network-association/sign-up

(96)

Interreg Europe (2018). Digital solutions in the field of cultural heritage  

(97)

  www.apex-project.eu

(98)

  https://pro.europeana.eu/post/presenting-the-opening-up-historiana-project

(99)

 See paper on the challenges and opportunities presented by the COVID-19 pandemic for the heritage world by Europa Nostra

(100)

Targeted consultation and DCHE workshop of November 2019.

(101)

See footnote 10.

(102)

European Commission (2017). Special Eurobarometer 466 on cultural heritage.

(103)

  https://www.europeana.eu/en/europeana-classroom

(104)

  https://pro.europeana.eu/post/making-your-collections-fit-for-education-a-case-study-from-euroclio

(105)

See also NUMERIC study

(106)

The statistics in this table were obtained from the Eurostat database . The indicator codes below provide a reference to the specific dataset where each indicator can be found.

(107)

Eurostat indicator code: isoc_ci_in_h.

(108)

Eurostat indicator code: isoc_ci_ifp_iu.

(109)

Eurostat indicator code: isoc_ec_ibuy.

(110)

Eurostat indicator code: cult_emp_age.

(111)

Eurostat indicator code: cult_emp_art.

(112)

Eurostat indicator code: gov_10a_exp.

(113)

European Commission (2017). Special Eurobarometer 466 on cultural heritage.

(114)

See paper on the challenges and opportunities presented by the COVID-19 pandemic for the heritage world by Europa Nostra

(115)

See responses to the online public consultation

(116)

See Motion for a European Parliament Resolution on artificial intelligence in education, culture and the audiovisual sector (2020/2017(INI))

(117)

See responses to the online consultation. Additional uses indicated were automated transcription, automated translation and visual recognition.

(118)

Feedback from the online consultation.

(119)

Consolidated Report 2018.

(120)

Workshop Brussels (January 2020).

(121)

See also responses to the online public consultation.

(122)

Workshop international organisations, December 2019

(123)

Workshop DCHE, November 2019 Luxembourg , responses to the online public consultation

(124)

Digital Cultural Heritage Roadmap for Preservation: https://www.dchrp.eu/

(125)

Particularly in the workshop on digital transformation, Brussels (January 2020)

(126)

See paper on the challenges and opportunities presented by the COVID-19 pandemic for the heritage world by Europa Nostra

(127)

Workshop Brussels ( January 2020)

(128)

  https://www.vi-mm.eu/wp-content/uploads/2016/12/ViMM-Manifesto-Revised-Final-Revised-19-November.pdf

(129)

Presentation of digital content, communication and outreach (of audience analysis and user-centric methodologies) were additionally referenced by stakeholders.

(130)

  https://ec.europa.eu/digital-single-market/en/news/digital-europe-programme-summary-report-targeted-consultation-future-investment-europes-digital

(131)

Games are an example of a creative participation in culture using digital technologies, with commercial/economic and social impact.

(132)

See footnote 4.

(133)

  COM(2007) 242 final

(134)

Communication from the Commission to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions, Towards an integrated approach to cultural heritage for Europe, COM (2014) 477 final , 22 July 2014.

(135)

  A New European Agenda for Culture, COM (2018) 267 .

(136)

See Section 4 of the New Agenda for Culture

(137)

  https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:52018XG1221(01)

(138)

  https://op.europa.eu/en/publication-detail/-/publication/5a9c3144-80f1-11e9-9f05-01aa75ed71a1

(139)

Communication from the Commission to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions, ‘A Digital Agenda for Europe’, COM(2010)245 final, 19 May 2010

(140)

Communication from the Commission to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions, ‘A Digital Single Market Strategy for Europe’, COM(2015) 192 final, 6 May 2015

(141)

Communication from the Commission to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions, ‘Shaping Europe's digital future’, COM(2020) 67 final, 19 February 2020

(142)

Communication from the Commission to the European parliament, the Council, the European Economic and Social Committee and the Committee of the Regions ‘A European strategy for data’, COM/2020/66 final, 19 February 2020

(143)

https://ec.europa.eu/digital-single-market/en/news/eu-member-states-sign-cooperate-digitising-cultural-heritage

(144)

  Directive (EU) 2019/790 of the European Parliament and of the Council of April 2019 on copyright and related rights in the Digital Single Market and amending Directives 96/9/EC and 2001/29/EC, OJ L 130, 17.5.2019, p. 92–125

(145)

  http://www.eblida.org/Experts%20Groups%20papers/EGIL-papers/MoU-OOC.pdf

(146)

Workshop Brussels (January 2020)

(147)

Directive (EU) 2019/1024 of the European Parliament and of the Council of 20 June 2019 on open data and the re-use of public sector information, OJ L 172, 26.6.2019, p. 56–83

(148)

Evaluation of Europeana and orientations for its future development, following adoption of Council Conclusions by EYCS Council on 31/05/2016

(149)

See Consolidated Report 2018

(150)

  Council Conclusions of 31 May 2016 on the role of Europeana for the digital access, visibility and use of European cultural heritage

(151)

See also feedback on the roadmap

(152)

See national report (2015-2017) provided by Austria

(153)

A majority was reached also across stakeholder groups and sectors.

(154)

As underlined in the feedback to the public consultation

(155)

SG, EAC, DIGIT, RTD, Ref. Ares(2019)4357448 - 08/07/2019

(156)

OJ L 283, 29.10.2011, p. 39–45

(157)

  https://ec.europa.eu/digital-single-market/en/news/eu-member-states-sign-cooperate-digitising-cultural-heritage

(158)

Organised with the assistance of the contractor within the framework of the Study supporting the evaluation of the Recommendation

(159)

Organised with the assistance of the contractor within the framework of the Study supporting the evaluation of the Recommendation

(160)

Building on the information collected from previous workshops

(161)

Except for Gaelic