EUROPEAN COMMISSION

Brussels, 11.6.2019

SWD(2019) 203 final

COMMISSION STAFF WORKING DOCUMENT

Accompanying the document

Report from the Commission to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions

Ex Post evaluation of the 2017 European Capitals of Culture (Pafos and Aarhus)

{COM(2019) 266 final}

Table of contents

1.Introduction

2.Background to the intervention

3.Implementation / state of Play

4.Method

5.Analysis and answers to the evaluation questions

6.Conclusions

Annex 1: Procedural information

Annex 2: Stakeholder consultation

Annex 3: Methods and analytical models

1.Introduction

This document describes the methodology and findings of the ex-post evaluation of the European Capital of Culture (ECOC) Action for 2017.

Article 12 of Decision No 1622/2006/EC 1 requires that the Commission ensures the external and independent evaluation of the results of the ECOC from the previous year. The purpose of the yearly evaluation, which the European Commission carries out since 2008, is to assess the achievements of the Action in the relevant year, to establish a comprehensive understanding of the overall performance and to put forward conclusions, recommendations and lessons for future ECOC title-holders, applicants and EU institutions.

This working document summarizes the findings of the external evaluation of the implementation of the ECOC 2017, including the selection and monitoring procedures and the operational delivery by the two hosting cities, namely Pafos in Cyprus and Aarhus in Denmark 2 . The external evaluation examined how the two cities developed their respective applications and cultural programmes, how they delivered the planned initiatives throughout the year, and any cultural, general and long-term impact generated by the Action.

The evaluation constitutes a valuable opportunity to critically reconsider the past year with the intention of reshaping insights and broadening wisdom in the light of the experiences of the host cities.

The aim of the evaluation is not to lead to any change in the regulation governing the ECOC Action. Indeed those rules were changed in 2014 3  on the basis of an ad hoc interim evaluation of selection and monitoring procedures of European Capitals of Culture 4 , but the new rules only apply for cities designated as ECOC for the years from 2020 to 2033 5 . It is therefore impossible to use the outcomes of the evaluation of the two ECOC 2017 to draw conclusions on the new legal basis as this new Decision doesn't apply to these two ECOC title-holders.

2.Background to the intervention

2.1.The European Capital of Culture Action

The initial scheme of "the European City of Culture" was launched at intergovernmental level in 1985 6 . On the basis of this experience, Decision No 1419/1999/EC 7 established a Community Action for the ECOC event for the years 2005 to 2019. Member States were ranked in a chronological order of entitlement to host the event each year. Decision No 1419/1999/EC was replaced by Decision No 1622/2006/EC, which kept the principle of a chronological order of Member States but further refined the objectives of the Action and introduced new selection and monitoring arrangements. As mentioned in the previous paragraph, a new Decision was adopted by the European Parliament and the Council in April 2014, but cities which were designated as European Capitals of Culture for the years up to 2019 continue to be regulated by Decision No 1622/2006/EC. The 2014 Decision was amended in September 2017 to open the ECOC Action to European Free Trade Association countries which are parties to the Agreement on the European Economic Area 8 .

2.2.Objectives of the ECOC Action

The ECOC Action aims to highlight the richness and diversity of European cultures and the features they share, thereby promoting greater mutual understanding among European citizens, as well as to foster the contribution of culture to the long-term development of the cities. ECOC shall strive to foster cooperation between cultural operators, artists and cities in Europe, foster the participation in cultural activities of the citizens living in the city and surroundings while raising the interest of citizens from abroad, to be sustainable and to be an integral part of the long-term cultural and social development of the city.

The hierarchy of objectives presented in the table below is based on the objectives as stated in Decision No 1622/2006/EC, but it has been updated to reflect the content of the new legal basis for ECOC post-2019. The general and strategic objectives are taken directly from Article 2 of Decision No 445/2014/EU, with the operational objectives flowing logically from these. They are also informed by the selection criteria detailed in Article 5 of the 2014 Decision.

Table on ECOC hierarchy of objectives

3.Implementation / state of Play

3.1.The selection and monitoring of the European Capitals of Culture 2017

In accordance with Decision No 1622/2006/EC, Cyprus and Denmark were entitled to host the ECOC in 2017. The relevant managing authorities of the two countries (i.e. the Culture Ministry of Cyprus and the Culture Ministry of Denmark) run parallel competitions, which both started in 2011 with the publication of a call to trigger applications from interested cities.

The selection is then in two phases: a pre-selection phase (candidate cities are reduced to a short-list) followed by a selection phase (one city is recommended for the title). A panel of thirteen members – six of whom nominated by the Member State concerned and the other seven by European Union institutions and bodies (European Parliament, Council, Commission and Committee of the Regions) – examined the bids from candidate cities on the basis of the objectives and criteria laid down in the Decision.

In Cyprus, three cities (Limassol, Nicosia and Pafos) responded to the call for applications by its deadline of 31 October 2011. Nicosia and Pafos were preselected in December 2011. In September 2012, Pafos was recommended at the final selection meeting. In May 2013, the Council officially designated this city as ECOC 2017 in Cyprus on the basis of the panel's recommendation.

In Denmark, Aarhus and Søndeborg entered into the competition by the call’s deadline of 30 September 2011 and both were pre-selected in December 2011, with Aarhus winning over at the selection meeting held in Copenhagen in September 2012. Finally, in May 2013, the city was designated by the Council as ECOC 2017 in Denmark.

From their nomination onwards, Pafos and Aarhus were subjected to monitoring arrangements: the progress in the cities' preparations was monitored and guided by a panel composed of the seven independent experts appointed by the European Union institutions and bodies, which also checked compliance with the programme and commitments on the basis of which the cities had been selected. Pafos and Aarhus attended two formal monitoring meetings with the panel convened by the Commission, in November 2014 and April 2016.

During this monitoring phase, the panel recommended both cities to adequately involve all various groups of citizens and to ensure a proper legacy of the year. The main difficulty encountered by Aarhus was related to the requirement of having an EU dimension in their activities. For Pafos the securing of funds – ultimately leading to a budget reduction – and the capacity to implement the programme were key issues.

During the monitoring phase, the two cities introduced modifications into the programme described in their original applications, in response to a changing environment (in particular the budget drop in the case of Pafos) and to the recommendations made by the panel. The panel also visited Pafos where it found it appropriate, but decided not to pay any visit to Aarhus as preparatory work there progressed smoothly.

The monitoring process culminated with the panel making a positive recommendation to the Commission on awarding a €1.5m prize in honour of Melina Mercouri to both cities after the last monitoring meeting. The Melina Mercouri prize was therefore awarded and paid by the Commission (under the EU Creative Europe programme 9 ) to the two ECOC in the autumn 2016.

The sub-sections below describe the main features of the ECOC programmes of Pafos and Aarhus. The final report of the contract supporting the evaluation contains further information, including the way the programmes had a European dimension and involved citizens (as these are the main two criteria of the ECOC Action) as well as on the development of the applications and the governance and funding structures.

3.2.Pafos 2017

Pafos in Western Cyprus dates back to the Neolithic Period. The city and the country as a whole are at the crossroads of the Eastern Mediterranean making it a multicultural melting pot. Its geographical proximity to Lebanon, Turkey, Syria, Egypt, Palestine and Israel, as well as continental Europe characterises its culture. As a consequence, Pafos aspired to become the first European Capital of Culture that linked East and West. The mythological past of the city that gave birth to Aphrodite, the Goddess of love and beauty, makes Pafos a mystical destination with many related relics and monuments. As this chapter will show, its historical past was a key re-emerging topic of the ECOC.

With a population of only 35,000, Pafos is one of the smallest ECOC hosts with its size being a significant factor affecting how the ECOC was funded, delivered and benefitted the city. Another defining feature of Pafos is the number of tourists – with 3.9 million tourists visiting in and around the city in 2017. Half of those tourists were from the UK and a third from Russia 10 . Also, of significance is the number of ex-patriate residents from the UK living in the city (estimates range from 10-25% of the total population). Linked to the city’s size, the cultural sector in Pafos is relatively small, less developed than in most ECOC host cities and often relates to the city’s past. The city’s architecture is often seen as its main cultural ‘offer’ including the Tomb of the Kings, Mosaics, Castle and numerous Churches.

The central idea of the Pafos2017 Programme was based on the ancient tradition, when culture developed in open spaces. The ‘Open Culture Factory’ (a term featuring heavily in the bid) that formed an important part of the cultural programme promised to travel to all areas and communities in Pafos, to display the activities of the Pafos2017 programme and to create a common space of communication and cooperation for everyone. This central idea was not only about open spaces but also about openness in terms of tolerance, acceptance, encouragement and integration of different cultures, ideas and beliefs.

The aim of the ECOC in Pafos was expressed in its motto “Linking Continents – Bridging Cultures”. The motto expressed the need for interconnection and bridging the separated inhabitants of Pafos including permanent residents, visitors and immigrants. It also highlighted how important it was to interconnect the scattered areas of the city and to turn the entire province of Pafos into a common space shared by all its citizens, both literally and metaphorically. Finally, it stressed the need to bridge the differences between the Greek Cypriot and the Turkish Cypriot community through various projects.

Responsibility for the organization of the ECOC was entrusted to the Pafos2017, a Foundation set up by the Municipality of Pafos soon after the city was awarded ECOC status. Pafos2017 was a private non-profit organisation and was responsible for the overall control of the ECOC, its budget and the delivery of some of the key projects. Its Board of Directors took charge of key decision around resourcing, priorities and key projects and none of these directors had a direct political role. The Board also had the overall responsibility to ‘make sure the company is working towards its strategic goals, in an effective way, both within the allocated budget and transparency’.

While being as high as €23 million at the bid stage, the Pafos2017 budget was small, amounting a total of €8.5 million. Despite their lack of involvement in terms of governance and practical implementation of the cultural programme, the Cypriot national government was the biggest financial contributor with around €4.5 million from the Ministry of Education and Culture, making up almost 60% of the total budget. With the exception of a few shifts in the timeline, the planned and actual income from the local and regional government were fully aligned, and amounted to roughly €1 million or 10% of the total budget.

3.3.Aarhus 2017

Aarhus was founded in the 8th century as a fortified Viking settlement in a natural harbour at the mouth of a fjord. For centuries, the primary driver for growth was seaborne trade in agricultural products and by the 1200s, it was significant enough for a large cathedral to be built.

Today Aarhus is Denmark’s second-largest city with a population of 335,684 inhabitants and is the cultural and economic centre of the Central Denmark Region (Region Midtjylland). In the past ten years, Aarhus has added more than 15,000 new residents to its population and created 20,000 new jobs, mostly within the knowledge, service and innovation industries with many important companies having their headquarters there. It is also an important centre of research and home to Scandinavia’s largest university, Aarhus University Hospital and the INCUBA Science Park.

Aarhus is notable for its post-war musical history of jazz and rock music and is home to several annual music festivals. The city hosts the ARoS Aarhus Museum of Art, the Moesgaard Museum dedicated to archaeology and ethnography, the Aarhus Theatre and the Concert Hall amongst other venues. During one week in summer, the city centre and parts of the harbour are turned into one large festival (Aarhus Festuge) that attracts visitors from all over the country.

Aarhus’s ECOC application emerged as part of a bigger plan for the development of the city, which focussed on construction and infrastructure developments around the seafront, including “Dokk1”: a new public library and culture centre featuring artistic installations. The application also aimed to promote more cohesive governance within the Central Denmark Region (CDR), one of five regions created in 2017. This new region encompasses 19 municipalities with Aarhus as the largest city whilst the regional capital is Viborg.

Aarhus 2017 presented itself under the narrative “Let’s Rethink”. The overall aim was to change mind-sets through cultural experiences and to highlight three core values of sustainability, diversity and democracy. The cultural programme was structured around four seasons, each of which opened with an outdoor, large-scale MEGA event. Each season also featured three Full Moon events, on a smaller scale than the MEGA events.

The implementation and management of the event was carried out by an autonomous foundation: Fonden Aarhus 2017 (“Aarhus 2017 Foundation”), which became operational as soon as the city was nominated. The Foundation was composed of a Secretariat that dealt with planning and coordination of the programme, budget, communication and partnerships, and a Board that gave the overall direction to the project. The Board consisted of 13 members of which seven were political appointments and six represented civil society in the form of experts with strong links to cultural institutions and major businesses in the area.

To ensure that the ECOC maintained its regional dimension, a Regional Steering Group was appointed with representation from all the participating municipalities. Furthermore, as Patron for Aarhus 2017, the Danish Queen was directly involved, participating in a preparatory video and book preface, speaking at the Opening Ceremony and attending several events. Likewise, the Prime Minister and the Minister for Culture were actively involved.

The eventual budget of Aarhus 2017 was €61.9m, which was 93% of the total proposed in the application. The shortfall arose due to national government funding being €9m less than hoped at the application stage, although some of the shortfall was made good by increased funding from Aarhus Municipality, the CDR and foundations and sponsors. By the end of 2016, the group of sponsors consisted of 28 foundations and 116 companies. EU funding was received in the form of the Melina Mercouri Prize, as well as funding for specific projects from the Culture and MEDIA sub-programmes within Creative Europe.

4.Method

In order for results to be comparable with previous evaluations, the methodology for this evaluation closely followed the approach adopted in previous assessments of the Action 11 , with focus being placed on research at city level and, in particular, on the gathering of data and stakeholders' views from both Pafos and Aarhus. The main evaluation sources can be identified as follows:

-EU level literature: higher level EU policy and legislative briefings, papers, decisions and other documents relating to ECOC – such as reports of the selection and monitoring panels – as well as academic research concerning the ECOC Action and the role of culture in the development of cities;

-ECOC level literature from Pafos and Aarhus: original bids and applications, internal reports linked to the application, monitoring or evaluation processes, as well as other pieces of literature analysing the cultural programme itself;

-Quantitative data: where available, evidence linked to each ECOC was collected in relation to budgets and expenditures, projects' numbers and types, participation and audience figures as well as other pieces of quantitative data relevant to the assessment of the work and benefits of the ECOC in each city;

-Interviews with managing teams: those responsible for the direct operational design and management of the ECOC projects were interviewed both during 2016 and again in 2017 (12 interviews of the Aarhus team and 11 interviews for the Pafos team). Almost all of the key positions linked to the delivery agencies were involved in the interviews, including those linked to strategic development, marketing and communication, project implementation and financial management;

-Interviews with key stakeholders 12 : interviews were undertaken with stakeholders both directly or indirectly involved in the planning, management or delivery of the ECOC along with those more widely linked to the cultural, social, economic or political agenda of the host cities. Stakeholders included personnel working in cultural organizations, in city/region/State level administrations, in tourism and visitor agencies, in media organizations as well as in voluntary and community organizations;

-An open public consultation 13 , as announced in the road-map 14 : launched on 11 March 2018 and closed on 21 July 2018, it was open to all people and organisations. However, some questions ideally required the respondent to have visited the ECOC city in order to give a response.

The final report of the contract supporting the evaluation provides a detailed understanding of the 2017 ECOC Action and within this an assessment of the work and progress of Pafos and Aarhus. There are however issues to consider when assessing the strengths of the evidence base used for this study:

-Although both cities have undertaken some form of evaluation work themselves, part of the results of these studies were not yet available at the time of the evaluation. The European evaluation of the ECOC Action has used as much of this secondary information as possible, but could not benefit from its final results, especially for what concerns quantitative data;

-As already mentioned in previous evaluations, an ideal way of conducting this evaluation would entail a before ('baseline') study and an after-picture ('ex-post') study, instead of carrying out the latter alone; the impossibility of comparing the two studies affects the accuracy of the evaluation. Moreover, since the evaluation was undertaken during and shortly after the end of the ECOC time framework some of the effects of the programme had not manifested themselves entirely. Many stakeholders involved in the evaluation commented that the real impact of the ECOC on the city and its residents would take time to filter through. In this respect, the Commission once again highlights that budget 15 and timing 16 only allow an ex-post evaluation to take place and therefore only an after picture has been studied;

-The public consultation undertaken to gather wider views from individuals and organisations on the ECOC acts as an open, inclusive and accessible route through which people can feed in their thoughts and opinions into the ECOC evaluation. The consultation was self-selective meaning anyone could feed their opinions into the evaluation without having to be asked. However, a non-selective research method such as the open survey means that anyone (including those with a vested interest) can take part and influence the outcome of the survey relatively easily. In addition, as a consultation tool, the public consultation provided a relatively narrow set of findings as only 76 responses were received overall. This is perhaps understandable as completing the survey required the respondent to know (and ideally to have attended) ECOC activities in the cities. For this reason, the results of the survey have often been used as supportive and sometimes contextual and have only been used in conjunction with other sources of evidence;

-An ulterior consequence of the modest yearly budget allocated to the evaluation is the fact that the primary evidence data gathering tends to be more of qualitative than quantitative nature; while qualitative data still holds a great importance in the evaluation, the lack of diversity of data sources translates into a lesser dependability, for instance, in the process of proving the objective outcomes and impacts of ECOC on widening participation in culture.

5.Analysis and answers to the evaluation questions

This part of the report presents a diagnosis of the implementation of the ECOC Action in 2017. It draws together the results relating to the two ECOC 2017 and to a lesser extent the findings from previous ECOC evaluations.

All figures appearing in the following sub-sections have been directly drawn from the report of the external ex-post evaluation of the two ECOC 2017. The reader will find in the full text of the document 17 many more examples illustrating the conclusions presented below.

Very often, examples and figures can illustrate different points made under "relevance", "efficiency", "effectiveness", "sustainability", "EU added value" and "coherence", and are therefore sometimes repeated in various sub-sections. However, for the ease of the reading, repetitions have been limited to the minimum and the reader is invited to correlate some conclusions presented below with examples or figures that may have been given in previous paragraphs or may be given in following sub-sections.

5.1.Relevance

Findings from the 2017 ECOC evaluation show that the two title-holders have developed and implemented a cultural programme that is consistent with the EU Treaty and in particular Article 167 18 . Indeed, in 2017 the ECOC Action continued to help EU Member States promote cultural diversity as well as highlight common elements, and added to the "flowering" of their – and European – cultural scene. Indeed, as will be shown in following sub-sections, in the two cities, thousands of events took place on top of the usual cultural offer, covering all types of cultural and artistic disciplines and presenting various aspects of the European cultural diversity notably through co-operations with partners from other (European) countries.

Although both cities clearly understood the strategic objectives of the ECOC, the programme in Aarhus was more relevant to the EU Action objectives with an articulated and comprehensive set of objectives that directly related to all the objectives of the Decision while Pafos objectives (to re-connect the Greek and Turkish Cypriot communities, re-connect the separate parts of the city, strengthen citizen’s identification with the city and enter into a more frequent cultural exchange with Europe and other regions) were narrower in scope.

The European dimension in Pafos

Pafos encapsulated well enough a European dimension at project level both in terms of highlighting Cypriot culture to Europe and also in terms of bringing culture from other European countries to Cyprus. Under its motto “Linking Continents – Bridging Cultures”, Pafos 2017 had three main themes with a clear European resonance: 1) Myth and Religion (Pafos cultural heritage and historical background as the place of birth of civilisation and its evolution linked back to the European civilisation), 2) World Travellers (in reference to the interconnections of people, intercultural dialogue, influences and interactivity that characterises Pafos and Cyprus as a link between East and West) and 3) Stages of the future (which was about the problems of the modern world, technology, dreams and hopes, originality and initiatives).

The cultural programme had a series of projects that showed the European dimension – in terms both of highlighting Cypriot culture to Europe and bringing cultures from other European countries to Cyprus. Examples include the International Sculture Symposium, the visit by the Berlin Philarmonic or the European Music Forum, which all proved instrumental in prompting debates and encounters between local people and European and international artists. An aspect of the European dimension covered in the bid-book but that was less obvious in the actual delivery of the cultural programme was around bringing eastern and western Europe together. The bid-book mentioned a number of times about the unique geographical position of Cyprus and Pafos between eastern and western Europe and the city’s proximity to places such as the Lebanon, Turkey, Syria, Egypt, Palestine and Israel 19 . In reality, the cultural programme that was delivered had less emphasis on this aspect – although there were a few examples of individual artists coming from these countries.

The “City and Citizens” dimension in Pafos 

Such dimension was reflected in activities that took place across the city, with an emphasis on using open spaces. This was a strong contributing factor in mobilising all citizens – both local residents and the international expat community – to both re-discover the cultural heritage of their city; and connect them with the communities living in these different spaces. According to the Cultural Programme of Events documentation, around 70% of projects were delivered outside. To accommodate for this, the Municipality invested heavily (around 14 million euros) in re-developing key areas of the old city centre. These significant changes made to their city in the years leading up to the ECOC helped citizens of Pafos embrace the ECOC project from the start.

Evidence of this can be found in the survey of local residents carried out by the local evaluation. It reveals that the respondents’ perceptions on the value and contribution of the ECOC to the city of Pafos was high from the outset, and increased as the ECOC year progressed. This survey showed that 59.5% of respondents who attended ECOC events felt that the added value of the ECOC to the city was very positive and 32.7% thought it was positive. The same survey showed that even those local residents who did not attend ECOC events also thought the ECOC had high levels of added value with a total of 84% of responses saying the effects were either very positive or positive.

The European dimension in Aarhus

The objectives of Aarhus 2017 were reflected in a set of “core values”, “concepts” and “motivations” that determined the overall narrative and programming of the title year. One – important –motivation was the “European dimension”, which was reflected in different ways.

Firstly, there were performances by a diversity of European and international artists as well as exhibitions of international works, exploration of European cultural diversity and common history, and connections with European networks.

Secondly, to showcase the European cultural diversity and common history, the programme included events that addressed contemporary issues, such as the European migration crisis and Brexit. Key events included for example the Eutopia International Festival, which attracted artists from 23 different countries who performed a diversity of works from across Europe and beyond or the “Music Unites Europe” festival, which involved numerous intellectually-challenged artists from several different European countries.

The “City and Citizens” dimension in Aarhus

Aarhus2017 has a clear “City and citizens” dimension articulated under three of the four Motivations of their Strategic Business plan, namely ‘Cultural infrastructure’, ‘Soft City’ and ‘City and Citizens’. In practice, the programme of Aarhus2017 used the city space (and spaces across the region) in new ways, included using the newly-developed waterfront area in Aarhus in ways that were not previously possible.

It created new opportunities for citizens (with a special focus on children and young people) to participate in culture through different approaches from working with existing community groups and schools, to engaging individuals through local advertising and social media. Opportunities were also created for citizens to participate as performers. Finally, it had an important regional dimension, with some 200 events taking place throughout the Region of Central Denmark.

5.2.Efficiency

Overall, the report of the contractor concludes that the ECOC remains an efficient EU Action, providing good levels of benefits for the EU for relatively little EU investment. Indeed, the only direct contribution from the European Union, in the form of the €1,5m Melina Mercouri prize awarded to each ECOC, is dwarfed by the total amount of money invested by the host cities in designing and delivering the ECOC (the operational budgets were approximately €66.7m for Aarhus and € 8.5m for Pafos).

Without the initial EU impetus to support the ECOC, it is unlikely that the host cities would have invested anywhere near the amount of funding they did in connection with the ECOC title. The possibility of securing the title typically 20 stimulates cities – but also their respective regional and national public authorities – to invest much more heavily in their cultural offer than they would in the absence of the ECOC – both in terms of infrastructure (this was more the case in Aarhus than in Pafos) and expenditure on cultural events and operations. The independent evaluation concludes that this means the Action remains highly efficient in terms of returns from the Melina Mercouri prize.

On the other hand, the final report of the contract supporting the evaluation also highlights that delivering an ECOC is still very challenging at the city level from an efficiency point of view, due to the need to build cross-party political support, secure funding from diverse sources (including national government), build capacity for implementation and ensure artistic. In this respect, it is worth mentioning that Pafos had difficulties in securing the budget initially sought for. On average, in the delivery plan up to 60% of the projects’ budget was cut compared to the estimated budgets from the bid-book

Notwithstanding the impact of the reduced budget, the working arrangement of the Pafos 2017 team was efficient and seen by the external evaluators as a strong point of the programme. All staff members tended to have multiple responsibilities and roles and the team was successful in generating a lot of interest from volunteers who played a critical role in the programme's delivery. The small size of Pafos city meant that local people tended to feel responsible for its success and were prouder of it in a way that people in other larger cities with ECOC status might not experience to the same extent. This is evidenced by the high levels of participation in the cultural events by local residents as well as the high number (350) of citizens who decided to enrol in the ECOC Volunteer Programme.  However, the communications activities of Pafos did not prove effective in reaching a wide international audience beyond those international visitors and expatriates already present in the city. Pafos implemented a local evaluation, which provided useful results but was quite limited in scope and made no use of big data.

In Aarhus, the arrangements for governance and implementation proved efficient, drawing on the strong political support both in the city and across the region and despite difficulties arising from the departure of senior staff during the development phase 21 . This is reflected in the fact that the Foundation and its partners delivered a cultural programme of the intended scope, scale and quality, raised the necessary resources and generally fulfilled the expectations set for the ECOC. Moreover, the regional collaboration also proved effective, given that the ECOC was the first time that the different municipalities of the CDR had co-operated in the field of culture in this way. Indeed, it appears from the evaluation that the local and regional collaboration partners were actively involved throughout the planning process, provided important financial backing and hosted many events that showcased the varied culture and heritage of the region. A key factor here included the Regional Steering Group, which brought together the Heads of Culture from each municipality and the CDR (and has been continued after the ECOC year). Aarhus also implemented a comprehensive and effective local programme of evaluation and research, although the use of big data was limited.

Aarhus proved broadly successful in raising the resources promised in the application. Whilst the sum raised from some sources, e.g. in-kind sponsorship, was less than proposed, the shortfall was offset by increased from elsewhere. Overall, the award of the title led to considerable additional investment in culture of around €66.7m during the years 2013-18. Whilst the Melina Mercouri Prize added value, Aarhus has shown that, it is the award of the title rather than the additional EU funding, which is the trigger for such investment in large or medium-size cities.

Regarding the efficiency of the selection, monitoring and EU co-financing procedures, the experience of the 2017 has demonstrated some strengths and weaknesses. On the one hand, the selection and monitoring procedures, as well as the informal meetings with the panel prove valuable in giving impartial advice and support to the ECOC from highly experienced experts, some of whom have implemented previous ECOC. Moreover, the formal meetings provide an opportunity for the panel to hold the cities to account, in terms of respecting the promises made in their respective applications.

At the same time, the experience of Pafos shows the limits to the procedures set out in the 2006 Decision. More specifically, the procedures have not proved sufficiently robust to prevent a reduction of 60% in the funding committed to the Pafos ECOC. For that reason, the changes introduced by the 2014 Decision (to be applied to the 2020-33 titles) are to be welcomed. In particular, it is encouraging to note that, for the 2020-33 titles, the Melina Mercouri Prize will only be awarded if the title-holders honour the commitments made at the application stage, including maintaining the budget at a level capable of delivering a high-quality cultural programme in line with the application and the criteria.

5.3.Effectiveness

Ultimately, the effectiveness of the ECOC in achieving the objectives set for them (at EU level and local level) can only be determined in the long-run. That being said, the final report of the contract supporting the evaluation however shows that both cities have proved successful in enhancing the range, diversity and European dimension of the cultural offer in their respective cities during 2017.

Both presented cultural programmes that were more extensive, diverse, innovative and international compared to the cultural baseline offering in previous years. In total, the ECOC cultural programme in Pafos involved 168 separate projects, which emerged from both a bidding phase or came out more organically from national and international synergies and collaborations. In Aarhus, a total of 442 core projects were implemented, corresponding to 13,708 “event days”. Whilst some projects featured the development or repeat of activities and events in previous years, the majority of each cultural programme was genuinely additional. More European and other international artists were involved compared to previous years, some new works were created and performed or exhibited and spaces in both cities were used in new ways to host cultural events.

Both ECOC widened access to and participation in culture during 2017, although the evidence is stronger in Aarhus than in Pafos. Both ECOC have helped strengthen the cultural capacity of the local cultural and creative sectors and their links with other sectors. The ECOC has raised the international profile of Aarhus through culture, whilst in both cities the ECOC has helped make audiences for culture more international.

Effectiveness in delivering a cultural programme with a European dimension

The Aarhus cultural programme was of the scale and quality proposed in the original application and had a genuine European dimension, as shown in the paragraphs below. According to the evaluation, Aarhus 2017 has been successful in contributing to making the cultural offering of Aarhus and the CDR more European.

Indeed, the cultural programme featured performances by a diversity of international artists and exhibitions of international works, strengthened European networks and connections, European co-productions and residencies by European artists. According to data provided by rethinkIMPACTS 2017, 1,200 international artists contributed to the programme. Some 79% of projects featured an international partner and/or a cultural exchange within Europe.

The performance of Pafos against its objectives was limited by the reduction in its budget compared to the figure proposed in the application. Reflecting this, the size of the cultural programme was far less than originally proposed, although more ambitious than the yearly annual offering in a city with a low cultural base before the ECOC-year.

Another element of the Pafos 2017 European dimension was the ambition to promote togetherness (a European value) in order to bring north and south Cypriots together but the cultural programme was perhaps too small to effectively deal with this complex issue. It did “start a conversation” about coexisting at the individual level but did not make a significant impression at a higher level.

Effectiveness in reaching and engaging with local citizens, in targeting specific groups and in strengthening the local cultural and creative sectors

Both ECOC widened access to and participation in culture during 2017, although the evidence is stronger in Aarhus than in Pafos.

Aarhus attracted audiences of 3.3m and ensured that the total audience for culture in the city and the region were greater in 2017 than in 2016. The four MEGA events and the twelve Full Moon events attracted total audiences of around 1.35m. These audiences were additional to the baseline situation in previous years, since these events were new for the title-year. Audiences for both types of event exceeded the targets: for MEGA events, the average audience was 196,722 (against a target of 60,000) while it was 46,847 for the 12 Full Moon Events (target: 15,000).

Second, the external evaluation shows that Aarhus reached a high proportion of local citizens, with 98% of the city’s inhabitants aware of the ECOC while 60% of people in the city and 40% in the region attended at least one event. This was no doubt facilitated by the fact that 52% of events were free to attend. Of the respondents to the European-level public consultation, 68% reported that the Aarhus 2017 encouraged them to attend more cultural events. Moreover, 65% of respondents stated that the ECOC had encouraged them to attend a wider type or genre of cultural events in 2017.

Third, Aarhus 2017 increased the number of citizens involved as creators, performers and audiences, including children and young people compared to previous years. Most notably, some 6,000 people actively contributed to the “People’s Opening”. In advance of the title-year, some 10,000 residents of the region were consulted.

Finally, Aarhus 2017 involved many more citizens as volunteers in culture compared to previous years. The “Rethinkers” volunteer programme involved 4,535 volunteers in some capacity during 2017.

In Pafos, the collection of audience data was not comprehensive, making it difficult to grasp the full extent to which access was widened.

Despite this shortcoming, it nonetheless appears that Pafos 2017 was successful in developing the audience for culture. According to the local evaluation, the total number of audiences reached 207,250 – a figure to see in the context of a city with a population of around 35,000 inhabitants – and around half of stakeholders interviewed reported that the city had “become hungry for culture”, which had never been the case before. As an example, it emerges from the same local evaluation that 58.9% of ECOC participants attended more cultural events as the year progressed while for 30.2% their involvement stayed more or less the same. The local evaluation also reveals there was a good age and gender balance across the audience, a mix of residents from different parts of the city, including the international expatriate community living in Pafos and more traditional “sunseekers”, as well as a high degree of diversity amongst audiences, despite the lack of focus on the participation of disadvantaged groups.

Finally, the volunteer programme was another key factor in mobilising the citizens to actively engage with the programme: more than 350 local and international volunteers were mobilised for the practical implementation of ECOC activities.

Both ECOC also helped strengthen the cultural capacity of the local cultural and creative sectors and their links with other sectors. Aarhus 2017 developed the capacity of cultural operators in the city and the region through various ways. For example, according to the Foundation’s “Welcome Future” report, some 48% of projects aimed directly at developing skills and competences, in line with the “Soft city” motivation. There were also 100 small-scale cultural projects implemented by project managers aged under 35 years and with limited experience of implementing such projects 22 . It has also provided an increased sponsorship of culture (the total additional sponsorship attracted from corporate sponsors and foundations amounted to €14.9m) and brought positive economic benefits to the cultural and creative sectors. In Pafos, both the stakeholders’ interviews and the Open public consultation reported that the title developed the skills and capacity of local cultural operators. Around three quarters of stakeholders' interviewed revealed that, for most local artists and project coordinators, the ECOC was their first opportunity to get involved in projects at such a large scale, and to have such a large and international audience for their artistic products.

Effectiveness in reaching out to European and international audiences

According to the evaluation, the ECOC raised the international profile of Aarhus through culture, whilst in Pafos, the ECOC helped make audiences for culture more international.

Aarhus 2017 attracted considerable attention from national and international media and from social media, raised the on-line international profile of Aarhus and increased the number of domestic and international visitors to the city and the CDR during 2017. Data from the Foundation shows that there were 27,723 media mentions in 2017, of which 2,528 were international (with some 95% of media mentions of a positive). It also appears that Aarhus 2017 featured in media articles in 70 countries. Furthermore, from 2016 to 2017, the number of overnight stays in the CDR increased by 4.0% while the number of hotel bed-nights increased by 10.9% and the number of hotel bed-nights accounted for by foreign visitors increased by more than 21,000, equivalent to 8.4% and there was an increase of 24.4% in the number of cruise ship visitors arriving in Aarhus in 2017.

The efforts to promote Pafos 2017 internationally were quite limited and, as a result, awareness of the ECOC amongst visitors to Cyprus was very low. However, the ECOC was successful at attracting expatriates and foreign tourists once they had arrived in Pafos, with non-Cypriots accounting for almost 40% of audiences. This is significantly higher than other ECOC, where foreign visitors typically make up less than 10% of total audiences.

It is to be noted that Pafos 2017 was not so much about increasing the international profile of the city (which was already high before the ECOC year with many foreign tourists visiting the city or expats living in the city) as it was about changing it. This is evidenced by the fact that there were no press releases in the ECOC city or the international travel press or advertising in overseas locations to attract more overseas visitors to Pafos. The focus of Pafos 2017 was more about encouraging the local population and tourists already in the city to take part in the ECOC and the wider purpose was to increase the potential of the old city centre as a ‘cultural vector’ for tourism, in addition to the more ‘traditional beach tourism’ which Pafos has always been known for.

Effectiveness in ensuring sustainability

The two 2017 ECOC cities offer very different potential for the sustainability of their activities and of improved cultural governance.

Pafos will see some long-term benefits but little has been done to ensure these benefits are maximised. Whilst some of those effects might endure – and funding has been provided in both cities for legacy/continuation projects – this will rely on continued partnership working by the various stakeholders, since neither city is planning a specific legacy body.

Although the Pafos ECOC year generated a lot of interest and excitement during 2017, this was already diminishing in early 2018 as staff posts and projects stopped being funded. Soon after December 2017, projects had disappeared and momentum (particularly in terms of volunteers) was on a sharp decline. The lack of any attempt to have a legacy plan developed even during the year is a key weakness of the ECOC.

Because the cultural programme was relatively small it needed to have a long-term plan in place to make a meaningful step change in the cultural offer of the city. If the ECOC had been awarded to another city with a larger existing cultural infrastructure, then the level of impact in the city may well have been higher than it actually was in Pafos.

On a more positive note, Pafos 2017 also served as a catalyst for cultural infrastructure investments, with a range of new infrastructure developments and renovations including the Ibrahim’s Kahn, the Attikon and the Markideion theatre, and these new venues will continue working in the future, providing Pafos 2017 with some legacy.

For what concerns Aarhus, the ECOC project generated potential for long-term impact through the skills and experience gained by cultural operators, the involvement of citizens, increased audiences and greater international profile.

Planning for the legacy of Aarhus 2017 began before the title-year, including through a consultation process that took place from May 2016 to January 2017. The consultation consisted of meetings, interviews and workshops with more than 100 stakeholders. It culminated in the publication of a legacy document: “Our legacy: a new beginning”, which was published by the Foundation Board in March 2017. Given the early publication of this document, the focus was mostly on those effects that were likely to endure, rather than legacy plans per se.

In Aarhus, several indicators based on data from the Visit Aarhus Tourism Barometer shows a slowing in growth in tourist visits and cultural audiences after the title-year, as is the case for most title-holders. However, tourist visits and cultural audiences were still higher in the six months after the title-year (Jan-Jun 2018) compared to the same period in the year before the ECOC (2016), which suggests the potential for long-term impact compared to the “baseline” situation prior to the ECOC.

The continuation of key projects after the end of the ECOC year, such as the ARoS Triennial and the Children’s International Festival, offers the potential for long-term impact, as does the continuation of the volunteer programme. Whilst much of the experience and expertise will be retained in the city and the CDR, the lack of a specific legacy body and an overall continuation strategy risks a loss of momentum. More promisingly, the potential for continued regional collaboration in the field of culture look promising, given the new structures put in place.

5.4.Coherence

The ECOC Action is coherent and complementary to the Creative Europe Programme in that it promotes the objectives of Creative Europe and is distinct from the other activities supported by the programme. Aarhus and Pafos also made use of funding from Creative Europe to support projects in their cultural programmes. Funding from the Europe for Citizens programme also helped Aarhus continue its volunteer programme in 2018.

The ECOC Action is also coherent with and complementary to the European Structural and Investment Funds, depending on the context of each city holding the title. In the case of Pafos, ERDF was used in the years leading up to 2017 to co-finance essential investments in the refurbishment of the city centre and renovation of key venues, such as a theatre and a cinema. Those investments thus supported the successful implementation of the ECOC cultural programme, whilst the ECOC helped “valorise” the investments by ensuring that the venues were used and given greater profile.

EU cross-programme funding also helped maximize the results of the ECOC Action. For instance, Aarhus 2017 featured some links to other EU programmes and initiatives, albeit modest. The Creative Europe (MEDIA sub-programme) co-funded the DICTE III crime and drama series, which is set in Aarhus and featured in the cultural programme of Aarhus 2017. 23 The European Capital of Volunteering, which is not an EU title but is supported by funding from the EU’s Europe for Citizens programme, was awarded to Aarhus for 2018. This provided additional impetus and profile to the continuation of the ECOC volunteer programme.

5.5.EU added value

As already mentioned and illustrated above, the ECOC Action has achieved an impact that would not have arisen through the actions of Member States alone.

The designation of Pafos and Aarhus as ECOC has attracted benefits that would have been unlikely to arise to the same extent in the absence of the Action. Indeed, the evidence presented in the chapters here above suggest that the ECOC, as an EU initiative and an EU branding, provides the stimulus for stakeholders to commit resources and effort to a shared vision and collaborative programme at a much greater scale than would otherwise happen. The EU added value consists to a great extent in the force of the 'brand' itself to act as a significant generator of interest from stakeholders not only from the city and but also from far beyond.

This is evidenced, for example, by the creation of dedicated organisations to implement the ECOC and the allocation of resources that are additional to the “mainstream” funding for cultural activities in the cities and regions concerned.

Regarding the visibility of the EU, both title-holders gave full prominence to the title of “European Capital of Culture” in their communication and promotional materials. Aarhus also made extensive use of the EU logo and gave prominence to the fact that the ECOC is an EU Action.

The results consistently achieved by the Action through the years establish that its models and mechanisms – for example in terms of building a solid governance for the delivery of the ECOC, increasing the capacity of local cultural organizations or attracting projects' ideas from local residents – can be applied not only within one individual Member State that has its administrative and cultural specificities but in any Member States finally confirming its EU added value.

6.Conclusions

The Commission concludes that the ECOC Action is highly valued by the hosting cities that can obtain positive impacts during the year, as well as during the preparation phase. The action also remains relevant at EU level. However, the long-term impacts of the two ECOC 2017 cannot be assessed at this stage, as it is too early after the implementation of the ECOC year.

The Commission also concludes that the programmes implemented by the two 2017 title-holders were consistent with the objectives of the ECOC Action:

-They reflected its European dimension: especially in Aarhus with for example 79% of projects featuring an international partner and/or a cultural exchange within Europe while Pafos 2017 was successful in mobilising its European and international expats and in reaching out its foreign visitors;

-They involved local residents and stakeholders: in Aarhus, the volunteer programme was one of the largest of any ECOC to date with 4 535 citizens volunteering for the year, Pafos did a good job in terms of involving the city and its residents in the actual delivery of the programme, with for example 59% of the audience coming from the city itself and local hotels donating 3,000 free nights to visiting artists;

-They widened access to and participation in culture during 2017, although the evidence is stronger in Aarhus than in Pafos; Both ECOC also helped strengthen the cultural capacity of the local cultural and creative sectors and their links with other sectors: almost half of the Aarhus 2017 projects aimed directly at developing skills and competences, in line with the “Soft city” motivation. In Pafos, the coordinators of individual ECOC projects and their local volunteers increased their skills as they had a lot more responsibilities when it came to the implementation of their activities compared with other ECOC;

-The ECOC raised the international profile of Aarhus through culture, whilst in both cities the ECOC helped make audiences for culture more international;

-Both ECOC year may lead to some legacies both physical (in the form of new or refurbished cultural and logistic infrastructures, as it happened in both cities) or intangible (by using open spaces, as Pafos did) although a proper legacy planning is lacking in Pafos (the legacy issue is exacerbated in Pafos because the cultural sector does not have enough scale to automatically ‘take on’ new and additional cultural projects without the injection of funding and interest an ECOC year brings). Furthermore, in Aarhus, the ECOC was the first time that the different municipalities of the CDR had co-operated in the field of culture in this way, and this collaboration is due to continue.

The Commission also considers that the scope and length of the yearly evaluation it carries out in accordance with Article 12 of Decision No 1622/2006/EC does not allow consideration of the long-term impact of ECOC. It is therefore willing to review such impact as part of the evaluation exercise foreseen in Article 16 of Decision No 2014/445/EU. This evaluation “shall focus on placing all past European Capitals of Culture in a European context, allowing comparisons to be drawn and useful lessons to be learned for future European Capitals of Culture, as well as for all European cities”. Building on the results of the study published in December 2017 on "European Capitals of Culture: Success Strategies and Long-Term Effects" 24 and will make it possible to analyse longer-term impact.

Annex 1: Procedural information

1.Lead DG, Decide Planning/CWP references

The evaluation was led by Directorate General for Education, Youth, Sport and Culture (DG EAC). It is included in the Work Programme of Creative Europe for 2017 and in the Agenda Planning with the reference  EAC - PLAN/2017/1264.

2.Organisation and timing

The evaluation was supported by an external and independent evaluator, under a service contract. The service contract was implemented via a Framework Contract with reopening of competition and in accordance to the Financial Rules Applicable to the General Budget of the Union 25 and its Rules of Application 26 .

The evaluation roadmap was adopted on 24 May 2017 27 .

According to the roadmap, a Steering Committee including staff from the European Commission’s Directorate General for Education, Youth, Sport and Culture and Secretariat General was established in May 2017. The Steering Committee met in four occasions: to prepare the Terms of Reference (ultimately sent to contractors in July 2017), to approve the Inception Report in January 2018, to discuss the draft final report in July 2018 and to approve the final report in October 2018. Extensive correspondence between the Steering Committee members was held in between the meetings to follow-up on the evaluation.

3.Exception to the better regulation guidelines

No exception.

4.Evidence, sources and quality

The 2017 evaluation of the ECOC used a series of data sets to inform its findings. The main ones being:

-Interviews with over 67 stakeholders directly and indirectly involved in the planning, operation and delivery of the two ECOC programmes;

-A literature review of ECOC and European level information on the two ECOC including application/ bid information, EC Committee reports, cultural programme brochures, web sites and news articles. The ECOC's own external evaluations have also been used to inform the European evaluation process;

-An Open public consultation gathering the view from 76 respondents;

-Together, the above evidence base provides the evaluation with a valid and rounded set of data to inform the views on the main aspects of the ECOC evaluation including efficiency, effectiveness, sustainability and relevance. This view has been informed by:

·The scale of the consultation exercise. Around 150 individuals have fed their views and opinions into the evaluation process either through face to face interviews, telephone interviews or through;

·The nature of the consultation exercise. The evaluators were keen to consult with those who had a more indirect and external view of the two ECOC. These stakeholders including journalists, those not directly benefitting from the ECOC (e.g. rejected projects) as well as those working in the wider cultural policy agenda at regional and city level. This ensures the evaluation is not simply based on those who benefitted the most from the ECOC.

Annex 2: Stakeholder consultation

1.     Methodology used 

The consultation activities described in this synopsis report were conducted in the context of the evaluation of the European Capital of Culture (ECOC) 2017. The consultations sought to collate information and stakeholders’ views on the evaluation criteria (relevance, effectiveness, efficiency, sustainability, coherence and EU added value). The relevant stakeholders were mapped at the early stage of the evaluation and are described below.

2.    Overview of consultation methods and consultees

The stakeholder consultation methods used consisted of in-depth interviews and an open public consultation (OPC). Evidence from the consultations was triangulated evidence from other sources, including documentation and data linked to the ECOC, as well as data from other sources, such as tourist bodies and national statistics. The table below provides an overview of the types of stakeholders concluded via each method.

In-depth interviews

Two main types of in-depth interviews were undertaken:

§Interviews with managing teams: those responsible for the day-to-day design and delivery of the ECOC were interviewed in each city during city visits in late 2016 (i.e. during the host year) and in spring 2017. Almost all of the key individuals within the delivery agencies were interviewed including those linked to strategic development, marketing and communication, project implementation and financial management.

§Interviews with key stakeholders: mainly face-to-face interviews were undertaken with stakeholders both directly and indirectly involved in either the planning or delivery of the ECOC along with those more widely linked to the cultural, social, economic or political agenda of the host cities. Stakeholders included those working in cultural organisations, city/regional/national administrations, tourism and visitor agencies, media organisations as well as voluntary and community organisations. Managers of individual projects and activities supported through the ECOC Action that made up the cultural programme of each city were also interviewed.

The table below specifies the number of each type of interview.

Open Public Consultation

The OPC was launched on 11 March 2018 and closed on 21 July 2018. It was open to all people and organisations. However, some questions ideally required the respondent to have visited the ECOC city in order to give a response. The OPC received a total of 76 responses from across Europe. A number of respondents attended both ECOC.

The public consultation gathered a large number of responses from individuals, responding in their own personal capacity 84% (N=64), whilst 16% (N=12) responded in their professional capacity on behalf of an organisation.

The largest group of respondents were local residents living in Pafos, 36% (N=23), whilst less than half of this number, 16% (N=10), were local residents living in Aarhus. A relatively high number were citizens responding from elsewhere in Cyprus: 22% (N=14), in comparison to those responding from elsewhere in Denmark: 2% (N=1). People outside of Denmark and Cyprus constituted 25% (N=16) of those who responded.

Among the respondents who replied in their professional capacity, the largest number of respondents (N=4) represented their local/regional or national government in Denmark, whilst no one from Cyprus responded in this capacity. Representatives from a national organisation in Cyprus and a local organisation in Pafos were both the second most common type of respondent (N=2 for both).

A large number of respondents (N=32) had heard of the ECOC Action, as well as Pafos 2017 and Aarhus 2017. Some 15% (N=11) had only heard of the ECOC Action.

Most respondents (N=38) attended a cultural activity in Pafos during the 2017 ECOC, whilst 22% (N=17) attended an event in Aarhus. A number of respondents attended an activity both in Pafos and in Aarhus (N=11), whilst only 13% (N=10) of respondents had not attended a cultural event in either city. The majority of respondents (N=38) attended more than 12 activities during 2017, whilst only 17% (N=11) only attended 1-3 activities.

3.    Main topics and viewpoints emerging

3.1    Relevance

Neither the OPC nor the in-depth interviews directly requested consultees to provide an opinion on questions of relevance.

3.2    Effectiveness

Both the OPC and the in-depth interviews gathered views on the effectiveness of the ECOC against the four specific objectives (SO) of the ECOC Action

Aarhus

The results of both types of consultation activity were broadly similar regarding the effectiveness of Aarhus ECOC, although the in-depth interviewees tended to be slightly more positive than the respondents to the OPC were.

SO1: Enhance the range, diversity and European dimension of the cultural offer in cities, including through transnational co-operation

In Aarhus, the interviewees generally agreed that the ECOC cultural programme was of high quality and that it was of greater scale and had more of a European dimension compared to the city’s baseline offering in previous years. This finding was supported by the OPC respondents. Some 73% felt that the quality of the cultural programme was very strong or strong and only 3% felt it was weak or very weak. Some 66% reported that the number and distribution of cultural projects was strong or very strong, whilst only 3% felt it was weak. Some 64% stated that the type and scope of cultural projects was strong or very strong in Aarhus and only 6% reported it was weak or very weak.

There was a consensus amongst the in-depth interviewees that the cultural offer of the city and the region would be better in future years as a result the ECOC. This was supported by a majority of respondents to the OPC; some 52% reported that the ECOC had a high or very high impact on Aarhus in terms of increasing the range and diversity of the cultural offered by the city. Only 3% of OPC respondents stated that there was no impact.

SO2: Widen access to and participation in culture

In Aarhus, the consultees agreed that the ECOC involved more citizens (including children) as creators, performers and audiences. They tended to report that the ECOC had expanded existing audiences and created new ones. This finding was supported by the OPC respondents in terms of their own attendance at cultural events, with 68% reporting that the ECOC encouraged them to attend more cultural events in 2017. A similar proportion of OPC respondents 65% also reported that the ECOC encouraged them to attend a wider type or genre of culture in 2017 than in previous years.

However, the OPC respondents were slightly less positive about the effectiveness of the Aarhus ECOC in increasing access to culture for people in general. Some 45% reported that the ECOC had a high or very high impact on the number of people accessing culture, whilst another 26% reported a moderate impact. Similarly, 36% believed the ECOC had a high or very high impact on introducing more people in the city to culture who do not normally access it, with another 32% reporting a moderate impact.

SO3: Strengthen the capacity of the cultural and creative sector and its links with other sectors

There was a consensus amongst consultees that the ECOC helped strengthen the cultural capacity of the local cultural and creative sectors and their links with other sectors. This was broadly supported by the OPC respondents: 55% reported that the ECOC had a high or very high impact in terms of strengthening the city’s capacity in the cultural sector, whilst another 19% reported a moderate impact. Similarly, 63% of OPC respondents reported that the ECOC had a high impact on helping local cultural operators to co-operate with European partners and only 3% felt it had no impact.

SO4: Raise the international profile of cities through culture

In Aarhus, consultees also reported that the ECOC raised the international profile of the city through culture. They tended to highlight the increase in international tourist visits, the attention given by the international media and the response from social media users.

This finding was supported by the majority of respondents to the OPC consultation. In total, 80% of respondents believed that the ECOC had a moderate to a very high impact on the number of international visitors (of which 29% reported a very high impact). Similarly, 71% of respondents saw the ECOC as having a moderate to very high impact on increasing the number of Danish visitors to the city. Only 6% believed that the ECOC did not increase the number of visitors.

The OPC provided a slightly contradictory finding on the question of whether the ECOC had encouraged them to visit Aarhus more often in 2017: only 48% reported that it had. However, this finding is perhaps not reliable as includes local residents as well as residents of other parts of Denmark and of other countries.

Pafos

The results of Pafos were similar across the different consultation methods although the results of the OPC were more positive.

SO1: Enhance the range, diversity and European dimension of the cultural offer in cities, including through transnational co-operation

Stakeholders in Pafos general felt that their ECOC increased the range and diversity of the cultural offer mainly because the cultural offer in the city was comparatively limited prior to the ECOC year. Stakeholders felt that the quality of the offer was particularly limited as there were very few high profile artists, venues or other infrastructure in place pre- 2017. Through additional funding, larger audiences and the introduction of new producers and artists (often from Nicosia and elsewhere in Cyprus) the artistic quality increased during the ECOC year. The OPC survey backed this up by saying that 58.9% of ECOC participants felt that cultural activity in the city had increased significantly as a consequence of the ECOC status. Further evidence from the open public consultation shows that 73% of respondents felt that the ECOC year had seen an increase in the number and distribution of the cultural projects. 84% of respondents to the same survey state that the ECOC had encouraged them to attend more and a wider type of culture in 2017 than before.

Both interviews and the OPC results generally supported the view that Pafos had a strong European dimension although the OPC results were generally more positive in this respect. Stakeholders highlighted a number of examples of projects that promoted a European dimension (e.g. the International Sculpture Symposium) and also highlighted ways in which the Foundation promoted the importance of projects having a European dimension. The OPC also gave further and very positive insight on people’s opinions around the European Dimension. 52% of respondents stated that they felt the ECOC had either had a very high or high impact on promoting European cultures compared to 20% for Aarhus. In addition, 59% of respondents felt that the ECOC had helped promote local cultures from the host city compared to 37% in Aarhus.

SO2: Widen access to and participation in culture

Stakeholders and the OPC were generally positive about the ECOC widening participation in culture although they showed little knowledge of the ECOC specifically targeting harder to reach groups who traditionally did not access culture. Some of the culture took place in the open air which helped to open up access to all of the communities as there was no ticketing nor many event taking place ‘behind closed doors’.

Results from the public consultation show that the ECOC had a more positive effect on encouraging people to attend more cultural events in 2017. 80% of respondents said that the ECOC encouraged them to attend more culture (compared to 68% in Aarhus). In addition, 84% of respondents said that the ECOC encouraged them to attend a wider type or genre of cultural events in 2017 and 76% said that the ECOC encouraged them to visit the city more often. Finally, 59% of respondents to the open survey said that the ECOC in Pafos had increased the number of people accessing culture (by a ‘very high’ or ‘high’ impact).

SO3: Strengthen the capacity of the cultural and creative sector and its links with other sectors

There was a consensus amongst all consultees that the capacity of the sector had benefitted greatly from the ECOC. Developing skills and capacity came out strongly as a direct benefit of the ECOC. The ECOC open public consultation shows that 87% of respondents felt that ECOC had had some positive effect on building the cultural capacity of the sector in the city with 60% stating that the impact was either very high or high. Around three quarters of stakeholders' interviews revealed that, for most local artists and project coordinators, the ECOC was their first opportunity to get involved in projects at such a large scale, and to have such a large and international audience for their artistic products.

SO4: Raise the international profile of cities through culture

There was less agreement on the level to which the ECOC raised the international profile of Pafos. Stakeholders were generally in agreement with one another by saying that the focus of Pafos2017 was more about encouraging the local population and tourists already in the city to take part in the ECOC. They said that one of the key objectives of the “Open Air Factory” was to reconnect the local citizens with the cultural and historical heritage of Pafos, in particular in the old city centre. According to all stakeholders linked to the visitor economy interviewed as part of this evaluation, the wider purpose of attracting more visitors to the old city centre was to increase its potential as a ‘cultural vector’ for tourism, in addition to the more ‘traditional beach tourism’ which Pafos has always been known for. Therefore they felt that actual foreign visitor numbers had not increased due to the ECOC.

Despite the above, the open public consultation shows that a city having ECOC status does sometimes affect the likelihood of a person wanting to visit the city. The survey showed that 42% of respondents said that Pafos having ECOC status would mean they would be ‘much more’ likely to visit the city and a further 45% said that they would be ‘slightly more’ likely to visit the city. Only 13% said that having ECOC status would make no difference in their decision to visit Pafos. The public consultation also found that 58% of respondents thought that the ECOC had either a very high or a high impact on increasing international visitors to the city. The respondents to the public consultation were generally from the local city or region of Pafos, which suggests why their responses were generally positive in this respect. The results from the public consultation also show that a city having ECOC status can influence tourism numbers in a city as it becomes a contributing factor in their decision on which destination to choose.

3.3    Efficiency

The in-depth interviewees and the OPC respondents were asked to comment on various efficiency aspects of the ECOC. The in-depth interviewees were generally very knowledgeable about the implementation arrangements, given that they included the main stakeholders, including several that were independent of the bodies responsible for implementing the ECOC (e.g. media). It is not known how familiar the OPC respondents were regarding the implementation arrangements.

Aarhus

There was a consensus amongst the in-depth interviewees that the arrangements for governance and implementation proved efficient, drawing on the strong political support both in Aarhus and across the region and despite difficulties arising from the departure of senior staff during the development phase. This positive finding was supported by the OPC, with 57% of respondents for Aarhus stating that it performed strongly or very strongly. Only 15% responded negatively to this question. Similarly, the marketing information for Aarhus was well received with 57% of respondents stating that the information shared was strong or very strong, although 16% felt it was weak or very weak.

Representatives of the Aarhus ECOC reported that the EU-level selection and monitoring procedures and the informal meetings with the panel proved valuable in giving impartial advice and support to the ECOC from highly experienced experts. (OPC consultees were not asked to comment on this question).

Pafos

The interviewees and OPC respondents both felt that the Pafos ECOC was very efficient mainly because it delivered what they felt was a positive ECOC on a very small budget. They felt that the number of cultural activities, the quality of the cultural programme and the various benefits that were identified showed a very efficient ECOC when the size of the limited budget was considered. High levels of efficiency included activities being delivered outside (meaning now venue costs), the use of volunteers and also having limited high profile artists all added up to an efficiently run ECOC.

3.4    Sustainability

Aarhus

In Aarhus, consultees supported the view that the ECOC had generated potential for long-term impact through the skills and experience gained by cultural operators, involvement of citizens, increased audiences and greater international profile. It was also felt that the continuation of key projects offers the potential for long-term impact. The OPC respondents were also positive on this point, with 57% reporting that efforts to make the benefits and activities sustainable were strong or very strong and only 6% reporting them as weak or very weak. However, 20% did not know.

Some in-depth interviewees felt that the lack of a specific legacy body and an overall continuation strategy risks a loss of momentum. However, there was a consensus that the proposal for continued regional collaboration in the field of culture looks promising. (OPC respondents were not asked to comment on these questions).

Pafos

This issue is where the largest disagreement took place between the OPC and stakeholders, with the OPC being much more positive about sustainability. In general, around 40% of OPC respondents were positive about legacy arrangements whereas very few stakeholders (less than 10%) were generally positive about ECOC sustainability. The stakeholders pointed to a range of evidence including no sustainability plan, no contracts of ECOC staff running into 2018 and no planned projects in following years as the reasons why sustainability was poor. It is not known why respondents to the OPC were much more positive about this issue.

3.5    Coherence

Neither the OPC nor the in-depth interviews directly requested consultees to provide an opinion on questions of coherence.

3.6    EU added value

The OPC included some specific questions on the impact of the ECOC regarding awareness of other European cultures, European sentiment and the Commission’s oversight of the ECOC Action. The in-depth interviews did not require the consultees to comment on these issues meaning this sub-section is focussed on the OPC only.

Across all OPC respondents, 41% felt that the ECOC had a high or very impact on helping local people to be more aware of other European cultures outside their country, whilst another 37% felt it had a moderate impact. Fewer than 3% felt it had no impact. More respondents in Pafos (52%) than in Aarhus (20%) felt this impact was high or very high.

The majority (51%) of OPC respondents perceived the ECOC to have had high or very impact on helping promote local cultures to European visitors, whilst nearly a third (31%) reported a moderate impact. Only 1% reported no impact. Again, more respondents in Pafos (59%) than in Aarhus (37%) felt this impact was high or very high.

Only a minority of respondents (43%) felt that the ECOC had a high or very impact on helping local people feel more European, although another 30% reported a moderate impact. Only 7% reported no impact. Once again, more respondents in Pafos (49%) than in Aarhus (31%) felt this impact was high or very high.

Across the OPC, 93% of respondents were aware that the ECOC is an EU action overseen by the European Commission. This is perhaps unsurprising, as the OPC was hosted on the European Commission’s survey tool. 28 Some 86% of OPC respondents also believed that there is a benefit for the European Commission overseeing the ECOC action, whilst only 8% did not see such a benefit. The in-depth interviewees were not specifically asked to comment on this question, though none reported any particular dissatisfaction regarding the Commission’s oversight of the ECOC Action.

3.7    Use of the results of the consultations

The evidence from the OPC and in-depth interviews has been fully taken into account in the preparation of the final report. The in-depth interviews have helped provide the narrative for each ECOC by describing the activities that took place. The in-depth interviews have also provided evidence against the evaluation questions. This evidence has been triangulated against evidence from other sources, including the OPC, documentation and data linked to the ECOC and data from other sources, such as tourist bodies and national statistics.

The OPC has provided some useful quantitative data regarding the opinions of individuals and organisations on a range of questions. This data has helped confirm some of the findings supported by evidence from other sources. However, the results of the OPC have had relatively limited usefulness, as first, the consultation was self-selective, and, second, only 76 responses were received. For this reason, the OPC results of the survey have mostly been used as supportive and sometimes contextual and have only been used in conjunction with other sources of evidence.

Annex 3: Methods and analytical models

The figure below presents the overview of the methodology. A more detailed overview of the methodology and sources used for each of the evaluation questions is presented in the subsequent tables and text.

1. Overview of methodology and tasks of the evaluation:

Inception phase

Task 1 : Kick-off meeting

Task 2 : Initial consultations - List of candidates for interviews - Notes of first interviews

Task 3 : First visits to the cities ECOC

Task 4 : Desk research

Task 5 : Inception Report

Main Research phase

Task 6 : Public consultation

Task 7 : Initial Bulletin

Task 8 : Online survey of projects

Task 9 : Second visits to the cities

Task 10 : Additional interviews

Analysis and reporting phase

Task 11 : Information sharing and analysis

Task 12 : Interim Report

Task 13 : Factual check of the city reports

Task 14 : Final Report

Table 1.1 Evaluation questions: Relevance, EU added value and coherence

Table 1.2 Evaluation questions: Effectiveness

Table 1.3 Evaluation questions: Efficiency

Table 1.4 Evaluation Questions: Coherence

2.Details of the relevant evaluation tasks and methodologies:

Inception phase

The aim of the inception phase was to finalise the evaluation framework and research tools, address the project steering group's comments regarding the proposed method and collect background information on Pafos and Aarhus.

Initial consultations

The task consisted in identifying stakeholders to consult with, targeting people that are knowledgeable about the two cities, their cultural programmes and the achievements, lessons learnt and impact of the ECOC, including national, regional and local stakeholders. Discussion at the kick off meeting identified a need for strong triangulation and to gain opinions from a broader set of stakeholders.

First visits to the cities

Initial face to face visits were made, to Pafos on 28-29 November 2017 and Aarhus in December 2017.

Those consulted with during the initial visit to Pafos were:

·Christos Patsalides- Chairman of the Board of Directors for Pafos2017,

·Georgia Doetzer- Artistic Programme Director for Pafos2017,

·Terpsi Vasou- Financial and Administrative Director for Pafos2017.

Those consulted in Aarhus were:

·Rebecca Matthews, Chief Executive Officer Aarhus2017,

·Juliana Engberg, Programme Director Aarhus2017,

·Karin Buhl Slæggerup, Head of Partnerships & Development Aarhus2017,

·Rina Valeur, Head of Strategy & Operations Aarhus2017,

·Bent Sørensen, Head of Communications Aarhus2017.

Desk research including big data feasibility

This involved collection and review of literature, primary data from the delivery agency and available secondary data, as well as web and social media statistics. The list of the literature that was collected at the European level which was reviewed and assessed for usefulness for the sake of the evaluation can be found in the Report 29 . Much of the European level literature is helpful to either understand the policy drivers at EU level which support the two 2017 ECOC or provide background on the bidding and application stages for both 2017 cities (i.e. little of it provides information useful for the content, delivery and impact of the two programmes). More academic literature at the EU level linked to the ECOC was also searched for. Instead of looking at academic literature generally on 'European culture' and also literature before 2017, the contractor supporting the evaluation specifically looked at up to date (i.e. produced in 2017) pieces which linked to the ECOC Action. This search did not prove fruitful, although some academic literature at the city level was found.

To complement the above desk research task the contractor assessed the extent to which each ECOC had used big data as well as analysed web and social media to increase visibility and interest in the ECOC among country residents and internationally.

Open public consultation

The Open public consultation proved to be a source of complementary qualitative data for both cities.

Second visits to the cities

With a clear view emerging from desk research and project survey as to the overall relevance, efficiency and effectiveness of each ECOC, the contractor undertook a second three day visit to each city. The visit focused on conducting in-depth interviews with the managing teams, their key stakeholders and cultural operators identified during the initial visits- please see the annex for more information.

During the visits to the cities described below, the contractors supporting the evaluation also sourced stakeholders for consultation who did not directly benefit or take part in the ECOC programme and who they felt would have an honest, impartial and unbiased view of how the programme was developed or implemented. While the aim was not to look for purely negative views, the contractors wanted to speak with stakeholder who, for example:

-did not deliver an ECOC project or take part in the cultural programme attached to the ECOC;

-were a member of the local and national press who had written either an honest or negative piece on the ECOC (as opposed to simply writing a press release which, for example, communicated the content and dates of the cultural programme);

-had applied for ECOC funding (to be part of the ECOC cultural programme) and were rejected;

-were involved in ECOC activity but were often vocal about a particular aspect (e.g. the quality of culture on offer, the lack of audiences, the lack of diversity of cultural content);

-were not directly involved in the cultural policy agenda in the cities but instead had an economic, social or environmental view on the ECOC.

The list of stakeholders consulted is provided in the annex to the Report 30 . This list was partly developed by the ECOC delivery teams but also through an internet search for stakeholders who work in the above fields.

With this in mind, in Pafos, the list of stakeholders consulted includes in Pafos:

·Officials from the City, the Chamber of Commerce, the Regional Board of Tourism and Tourism organisation,

·Cultural operators from various sectors (theatre, music,…),

·Professor at the University.

In Aarhus, it included:

·Sponsor companies,

·Cultural operators,

·Associate Professor,

·Official from other municipalities.

Building on the interviews undertaken in the first visits, the interviews with the delivery teams served four purposes: i) gathering further factual data and information about the cultural programme and its achievements; ii) identifying the "story" of the ECOC throughout its lifecycle, i.e. conception, application, development, delivery, legacy; iii) gaining a critical (albeit "insider") perspective on the relevance, efficiency, effectiveness and sustainability of the ECOC; iv) triangulating emerging findings, for example, those emerging from the desk research and project survey.

Interviews with other key stakeholders in the cities (such as municipality, Chamber of Commerce, commercial sponsors, media representatives, and cultural operators) were also essential to identifying the overall "story" of the ECOC from conception, through application, to development and delivery. These individuals offered an external, alternative and broader perspective on the ECOC.

Additional project and stakeholder interviews

These interviews sought to gather an in-depth understanding of the effect that ECOC had for the individuals involved, their organizations and the local culture sector as a whole. It also allowed to ensure that findings are based on consensus across a range of target groups as well as to identify key differences in the experience of different stakeholders. In the case of projects, the criteria for inclusion focused on whether it represents good practice in relation to at least one of the key dimensions of the evaluation, including (but not limited to) increased European cooperation, the effective targeting of key groups of citizens or neighbourhoods, audience development strategies, or legacy effects.

Analysis and reporting phase

Once all the information gathered, the experts fulfilled an analysis of the available information, drawing conclusions and triangulating data to ensure consistency and accuracy. The limitations of the data gathered have been explained in the report and in the Staff Working Document. After the first drafting exercise, the results were shared with the two ECOC for a factual check. Both the Pafos and Aarhus ECOC were asked to undertake a check on the completeness of data being used to address each evaluation topic. Their comments, where relevant, were included in the Final Report submitted by the contractor supporting the evaluation.

(1)      Decision No 1622/2006/EC of the European Parliament and of the Council of 24 October 2006 establishing a Community Action for the European Capital of Culture event for the years 2007 to 2019 (OJ L 304, 3.11.2006, p. 1), available at: - http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=OJ:L:2006:304:0001:0006:EN:PDF .

(2)      Full document available at: https://ec.europa.eu/programmes/creative-europe/sites/creative-europe/files/ecoc-2017-evaluation-en.pdf

(3)      Decision No 445/2014/EU of the European Parliament and of the Council of 16 April 2014 establishing a Union action for the European Capitals of Culture for the years 2020 to 2033 and repealing Decision No 1622/2006/EC (OJ L 132, 3.5. 2014, p. 1), available at:    - http://eur-lex.europa.eu/legal-content/EN/TXT/?uri=uriserv:OJ.L_.2014.132.01.0001.01.ENG This Decision, which covers the ECOC titles 2020 to 2033, retains the general structure and main elements of the previous Decision while introducing improvements to maximise the benefits of holding the title as well as taking part in the competitive process for all bidding cities and their citizens. Improvements include among others the introduction of more explicit and measureable criteria, the reinforcement of conditionality for the payment of the Melina Mercouri Prize and the obligation for the cities – instead of the Commission – to carry out the ex-post evaluation of the ECOC year.

(4)      Available at: https://ec.europa.eu/programmes/creative-europe/sites/creative-europe/files/library/2011-capitals-culture-assignment-report_en.pdf

(5)      Article 17 of Decision No 445/2014/EU.

(6)      Resolution of the Ministers responsible for Cultural Affairs, meeting within the Council, of 13 June 1985 concerning the annual event 'European City of Culture' (85/C 153/02), available at http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=OJ:C:1985:153:0002:0003:EN:PDF .

(7)      Decision 1419/1999/EC of the European Parliament and of the Council of 25 May 1999 establishing a Community action for the European Capital of Culture event for the years 2005 to 2019 (OJ L 166, 1.7.1999, p. 1). The Decision was amended by Decision No 649/2005/EC to integrate the Member States that joined the EU in 2004 (OJ L 117, 4.5.2005, p. 20). Both Decisions are available at:- http://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A31999D1419 - http://eur-lex.europa.eu/eli/dec/2005/649(1)/oj

(8)      Decision (EU) 2017/1545 of the European Parliament and of the Council of 13 September 2017 amending Decision No 445/2014/EU establishing a Union action for the European Capitals of Culture for the years 2020 to 2033 (OJ L 237, 15.9.2017, p. 1). Text available at: http://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32017D1545  

(9)      Regulation (EU) No 1295/2013 of the European Parliament and of the Council of 11 December 2013 establishing the Creative Europe Programme (2014 to 2020) and repealing Decisions No 1718/2006/EC, No 1855/2006/EC and No 1041/2009/EC (OJ L 347, 20.12.2013, p.221).

(10)       http://cyprus-mail.com/2017/12/10/paphos-tourism-figures-last-year/

(11)      See previous evaluation reports available at: https://ec.europa.eu/programmes/creative-europe/actions/capitals-culture_en  (table at the bottom of the document).

(12)      A total of 37 stakeholders for Pafos and 30 for Aarhus were interviewed.

(13)      The OPC received a total of 76 responses from across Europe. A number of respondents attended both the European Capital of Culture (ECOC) in Pafos and Aarhus, which explains why the 56 respondents for Pafos and the 31 respondents for Aarhus exceeds the total number of respondents.

(14)   https://ec.europa.eu/info/law/better-regulation/initiatives/ares-2017-2639855_en .

(15)      The budget allocated to the evaluation work (75 000 €) is proportionate to the low level of EU funding directly provided to the ECOC (i.e. an award based on a recommendation of the panel after the final monitoring meeting in the form of the €1.5m Melina Mercouri Prize).

(16)      Decision No 1622/2006/EC requires that the Commission conducts the evaluation immediately after the title year.

(17)      See footnote 5.

(18)      Available at: http://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A12008E167 .

(19)       https://issuu.com/pafos2017/docs/pafos2017_bidbook .

(20)      More information on the Cities objectives when applying can be found in Chapter 3.3 of a European Parliament study available at: www.europarl.europa.eu/RegData/etudes/etudes/join/2013/513985/IPOL-CULT_ET(2013)513985_EN.pdf .

(21) As expressed by the panel during the first monitoring meeting – see report at https://ec.europa.eu/programmes/creative-europe/sites/creative-europe/files/files/ecoc-2017-aarhus-monitoring_en.pdf .

(22)      Welcome Future, Short-term impact of European Capital of Culture Aarhus 2017, Aarhus 2017 Foundation.

(23)       http://www.aarhus2017.dk/en/programme/film-and-animation/dicte-3/  

(24)      See:      http://www.europarl.europa.eu/RegData/etudes/etudes/join/2013/513985/IPOL-CULT_ET(2013)513985_EN.pdf

(25)       http://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:02012R0966-20160101&from=EN

(26)       http://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:02012R1268-20160101&from=EN

(27) https://ec.europa.eu/info/law/better-regulation/initiatives/ares-2017-2639855_en

(28) https://ec.europa.eu/eusurvey/

(29)      See previous footnote.

(30)       hthttps://ec.europa.eu/programmes/creative-europe/sites/creative-europe/files/ecoc-2017-evaluation-en.pdf