EUROPEAN COMMISSION

Brussels, 25.4.2018

SWD(2018) 140 final

COMMISSION STAFF WORKING DOCUMENT

Accompanying the document

Report from the Commission to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions

Ex Post evaluation of the 2016 European Capitals of Culture (Donostia San Sebastian and Wroclaw)

{COM(2018) 235 final}

Table of contents

1.Introduction

2.Background to the intervention

3.Implementation / state of Play

4.Method

5.Analysis and answers to the evaluation questions

6.Conclusions

Annex 1: Procedural information

Annex 2: Stakeholder consultation

Annex 3: Methods and analytical models

1.Introduction

This document describes the methodology and findings of the ex-post evaluation of the European Capital of Culture (ECOC) Action for 2016.

Decision No 1622/2006/EC 1 requires, in Article 12, that the Commission ensures the external and independent evaluation of the results of the ECOC from the previous year. The purpose of the yearly evaluation, which the European Commission carries out since 2008, is to assess the achievements of the Action in the relevant year, to establish a comprehensive understanding of the overall performance and to put forward conclusions, recommendations and lessons for future ECOC title-holders, applicants and EU institutions.

The aim of this evaluation exercise is not to lead to any change in the regulation governing the ECOC Action. Indeed those rules were changed in 2014 2 on the basis of an ad hoc interim evaluation of selection and monitoring procedures of European Capitals of Culture 3 , but the new rules only apply in the case of cities which are designated as ECOC for the years from 2020 to 2033 4 . It is therefore impossible to use the outcomes of the evaluation of the two ECOC 2016 to draw conclusions on the new legal basis as this new Decision doesn't apply to these two ECOC title-holders.

This working document summarizes the findings of the external evaluation of the implementation of the ECOC 2016, including the selection and monitoring procedures and the operational delivery by the two hosting cities, namely Donostia San Sebastian in Spain and Wroclaw in Poland 5 . The external evaluation examined how the two cities developed their respective applications and cultural programmes, how they delivered the planned initiatives throughout the year, and any cultural, general and long-term impact generated by the Action. 

The evaluation constitutes a valuable opportunity to critically reconsider the past year with the intention of reshaping insights and broadening wisdom in the light of the experiences of the host cities.

2.Background to the intervention

2.1.The European Capital of Culture Action

The initial scheme of "the European City of Culture" was launched at intergovernmental level in 1985 6 . On the basis of this experience, Decision No 1419/1999/EC 7 established a Community Action for the ECOC event for the years 2005 to 2019. Member States were ranked in a chronological order of entitlement to host the event each year. Decision No 1419/1999/EC was replaced by Decision No 1622/2006/EC which kept the principle of a chronological order of Member States but further refined the objectives of the Action and introduced new selection and monitoring arrangements. As mentioned in previous paragraph, a new Decision was adopted by the European Parliament and the Council in April 2014, but cities which were designated as European Capitals of Culture for the years up to 2019 continue to be regulated by former Decision No 1622/2006/EC. The 2014 Decision was amended in September 2017 to open the ECOC Action to European Free Trade Association countries which are parties to the Agreement on the European Economic Area 8 .

2.2.Objectives of the ECOC Action

The ECOC Action aims to highlight the richness and diversity of European cultures and the features they share, thereby promoting greater mutual understanding among European citizens, as well as to foster the contribution of culture to the long-term development of the cities. ECOC shall strive to foster cooperation between cultural operators, artists and cities in Europe, foster the participation in cultural activities of the citizens living in the city and surroundings while raising the interest of citizens from abroad, to be sustainable and to be an integral part of the long-term cultural and social development of the city.

The hierarchy of objectives presented in the Table below is based on the objectives as stated in Decision No 1622/2006/EC, but it has been updated to reflect the content of the new legal basis for ECOC post-2019. The general and strategic objectives are taken directly from Article 2 of Decision No 445/2014/EU, with the operational objectives flowing logically from these. They are also informed by the selection criteria detailed in Article 5 of the 2014 Decision.

Table on ECOC hierarchy of objectives

3.Implementation / state of Play

3.1.The selection and monitoring of the European Capitals of Culture 2016

In accordance with Decision No 1622/2006/EC, Spain and Poland were entitled to host the ECOC in 2016. The two-phased selection (shortlisting and nomination) was carried out by a panel of thirteen members – six of whom nominated by the Member State concerned and the other seven by European Union institutions and bodies (European Parliament, Council, Commission and Committee of the Regions) – who examined the bids from candidate cities on the basis of the objectives and criteria laid down in the Decision.

The two Member States launched their respective competition in parallel in the summer 2009. Donostia San Sebastian was shortlisted in September 2010 against 14 other bidding cities in Spain, while Wroclaw was shortlisted in October 2010 against other 10 competitors in Poland. In June 2011, both cities were recommended by the panel to host the 2016 title, and were nominated by the Council of the European Union in May 2012.

After their nomination, Donostia San Sebastian and Wroclaw were subjected to monitoring arrangements: the progress in the cities' preparations was monitored and guided by a panel composed of the seven independent experts appointed by the European Union institutions and bodies, which also checked compliance with the programme and commitments on the basis of which the cities had been selected. Donostia San Sebastian and Wroclaw attended two formal monitoring meetings convened by the Commission, in October 2013 and March 2015.

During this monitoring phase, the panel recommended both cities to adequately involve minorities and all various groups of citizens and to address the requirement of encompassing the EU dimension in their activities. The main difficulties encountered by Wroclaw concerned the initial securing of funds (at the time of the first monitoring report only 37% of the funds had already been secured), while in Donostia San Sebastian local political changes that happened between 2011, 2015 and 2016 (the former for local and province elections and the latter for the Basque Government and Spanish Government) affected a smooth development of the project.

During the monitoring phase, the two cities introduced modifications into the programme described in their original applications, in response to a changing environment and to the recommendations made by the panel. The panel also visited the cities where it found it appropriate.

The monitoring process culminated with the panel making a positive recommendation to the Commission on awarding a €1.5m prize in honour of Melina Mercouri to both cities after the last monitoring meeting. The Melina Mercouri prize was therefore awarded and paid by the Commission (under the EU Creative Europe programme 9 ) to the two ECOC in the autumn 2015.

The sub-sections below describe the main features of the ECOC programmes of Donostia San Sebastian and Wroclaw. The final report of the contract supporting the evaluation contains further information, including on the European dimension of the programmes, on the way the programmes involved citizens as well as on the development of the applications and the governance and funding structures.

3.2.Donostia San Sebastian 2016

Donostia San Sebastian is a city of about 186,000 people and the capital of Gipuzkoa Province in the Basque Autonomous Community of Spain, close to the French border. A unique feature of the city is its very rich cultural offer, with the number of cultural projects taking place in it during a 'non-ECOC' year, estimated between 1,000-1,500 (i.e. performances, exhibitions etc.), attracting well over 1 million visitors per year. Despite having been a popular tourist destination ever since the beginning of the 20th century, the city has been deeply scarred by the ETA's terrorist activities (in Euskara: Euskadi Ta Askatasuna, Basque Homeland and Freedom). ETA was created in 1959 as a group for the promotion of the Basque Culture, and later developed as a paramilitary group with the objective of independence of the Basque Country.

Bearing such background in mind, Donostia San Sebastian's application was entitled "Culture for coexistence"; it aimed to help overcome the past violence engraved in the identity of the city and to stimulate a culture of respectful tolerance between people who "share places where multiple identities, values and objectives cohabit and constantly keep redefining one another" 10 .

The concept of coexistence was translated into three main "lighthouses of change" that structured the whole cultural programme of the ECOC year and aimed at establishing sustainable cooperation. The three "lighthouses" were as follows:

-"Lighthouse of Peace" explored a very relevant topic for the Basque country, which faced decades of terrorism. In addition, this topic was particularly relevant in the European context where the question of integration and coexistence is becoming more and more urgent. The concept behind this lighthouse was to support new forms of governance, promoting and improving dignity and human rights;

-"Lighthouse of Life" dealt with the concept of human existence, being a member of society and a part of the environment. This path included topics such as health, work and the environment. One focus was to allow citizens to reflect on how they are in relation with each other and the environment;

-"Lighthouse of Voices" was seen as a way to reach coexistence between people. The idea was to use effective communication as a virtual mechanism enabling people from different societies to interact among themselves. This lighthouse was primarily for activities related to communication with each other (i.e. 'voices') and therefore the main difference in emphasis from the lighthouse of peace was around the importance of talking and working with one another. Communication was through culture and included art expression, such as music, literature and visual art as they are one of the ingredients for a more pluralistic society. It represented the concept of cultural and linguistic minorities, a topic particularly relevant for the Basque Country.

Transversal lines (called the 'quays') were developed to allow a more in-depth view on the cultural activities that were proposed by the three lighthouses. These lines were in fact five methodologies – in the fields of sustainability of linguistic diversity, critical thought through contemporary art, citizens' engagement, connection between culture and technology, and the development of interactive situations in cultural spaces – which were meant to help all individual projects to achieve their own objectives.

The city of Donostia San Sebastian was the main driver behind the initial ECOC bid, although the regional government were also very much involved in the initial discussions and the eventual bid. The three public administrations of the Basque Country (Municipality Council, Territorial Administration and the Basque Government) came together to form a 'consortium' that ensured a joint ownership of key decisions (the ones that were linked to priorities, themes and major projects).

The main focus of the governance arrangements was a Foundation that was specifically set up to design, deliver and manage the entire ECOC programme. This was an arm's length organization that was partly set up to ensure a quick and 'flexible' response to various issues and was meant to be largely free of political influence although it had to deal with a comparatively complex and layered political structure and was dependant on it for some key decisions. This foundation was effectively at the heart of the ECOC and was responsible for all its financial, marketing, communication and cultural work on a day to day basis.

The total planned budget for the ECOC as set out in San Sebastian's original bid was €89 million. The actual budget during 2016 was €49.6 million, of which 27% came from the City, 26% from the Province, 25% from the Basque Government and 10% from the Spanish Ministry of Culture. The overall budget figure was 44% less than planned. The main reasons for the difference between the planned and actual income were highlighted as very tight pressure on public sector budgets, political pressure to prioritise budgets on other issues of public interest and the Foundation's perceived or actual lack of capacity to seek out private sector funding coupled with poor economic conditions.

3.3.Wroclaw 2016

Wroclaw is Poland's fourth largest city with a population of 636,000 inhabitants, situated in the South-West of the country (Lower Silesia), not far from the Czech Republic. The city has had an eventful history, changing hands between Bohemia, Poland, Hungary, Austria, Prussia, Germany and modern-day Poland. Until the Second World War, the city was known as "Breslau" and was part of Germany. After the war, the city became part of Poland under the terms of the Potsdam Agreement. Most of the remaining German population of 190,000 fled or were forcibly removed, to be replaced by ethnic Poles, most of whom were resettled from parts of eastern Poland that were incorporated into the Soviet Union. At that point, the city was (re-) named "Wroclaw". During the communist period (1945-89), Wroclaw became one of the centres of opposition to the communist regime. Since the end of the communist period, Wroclaw has developed a reputation as one of the most liberal cities in Poland, embracing for example public and street art.

This diversity of influences has given the city a varied architectural heritage, including Silesian Gothic and Baroque but also Modernism, and a strong multicultural character, notably in its religious diversity as well as a very rich cultural life.

Wroclaw's ECOC application emerged from the wider strategy of the City of Wroclaw, which emphasised the potential for urban development to be supported by prioritizing investments in cultural and sporting events and infrastructures.

"Metamorphoses of Culture" was the overall concept adopted, which served as a metaphor for both the historical transformation of the city (i.e. the upheaval at the end of the Second World War when the German population was replaced by the ancestors of today's inhabitants) and for contemporary processes of cultural and social change (including globalisation, immigration, EU enlargement and the growing role of digital communications). The Wroclaw 2016 application presented the city as a "space for beauty", and the intention was to "create spaces within which to restore the presence of beauty in public life and in daily habits", reflecting also the multicultural character of the city.

The programme was divided into eight artistic domains – Architecture, Film, Literature (connected with the UNESCO World Book Capital 2016), Music, Opera, Performance, Theatre, and Visual Arts – each of them led by a curator and featuring a number of events and projects.

The programme had four different stages:

-The "Wroclaw stage", focusing on dialogue between the city and its inhabitants;

-The "Lower Silesia stage", promoting regional participation in the project;

-The "Polish stage", a platform for collaborative artistic activities from around the country; and

-The "European and World stage", showcasing international cooperation between artists and creators of culture, highlighting the city's role in the European and world culture.

Wroclaw envisioned very clear goals, especially for the longer term, which ranged from increasing awareness and recognition of the Wroclaw/Lower Silesia's cultural identity, to creating public spaces for social activities and shaping of civic attitudes, to increasing the number of visiting tourists 11 .

Responsibility for the organization of the ECOC was entrusted to "IMPART 2016 Festival Office" ("Impart"), a new cultural institution of Wroclaw Municipality formed from the merger of "Wroclaw 2016" (the institution responsible for preparing the application for the title) with "IMPART Art Centre" (a cultural management institution with production resources). Impart was responsible for the overall organization and coordination of the ECOC programme, acted as the main contact point and developed promotional and marketing campaigns. The leadership of the ECOC was entrusted to the Director-General of Impart, whilst the developing process of the cultural programme was delegated to a Board of Curators, all of whom were recognized cultural operators in their respective artistic domains.

The eventual operating income of Wroclaw 2016 was €86.4m (as against €78.6m in the original bid), of which €34.4m (40%) was provided by the City of Wroclaw and €31.7m (37%) by the National Government. Around €7m was received from corporate sponsors, including the main sponsor KGHM, the Polish multinational corporation specialising in copper and silver production, or in the form of in-kind income.

Unlike some other ECOC, the body responsible for the ECOC cultural programme remained in existence beyond the title-year and still continues to play a key role in coordinating cultural activities in the city: some exhibitions, artistic residencies and micro-grants schemes initiated in the frame of the ECOC 2016 were continued after 2016. 

4.Method

In order for results to be comparable with previous evaluations, the methodology for this evaluation closely followed the approach adopted in previous assessments of the Action 12 , with focus being placed on research at city level and, in particular, on the gathering of data and stakeholders' views from both Donostia San Sebastian and Wroclaw. The main evaluation sources can be identified as follows:

-EU level literature: higher level EU policy and legislative briefings, papers, decisions and other documents relating to ECOC – such as reports of the selection and monitoring panels – as well as academic research concerning the ECOC Action and the role of culture in the development of cities;

-ECOC level literature from Donostia San Sebastian and Wroclaw: original bids and applications, internal reports linked to the application, monitoring or evaluation processes, as well as other pieces of literature analysing the cultural programme itself;

-Quantitative data: where available, evidence linked to each ECOC was collected in relation to budgets and expenditures, projects' numbers and types, participation and audience figures as well as other pieces of quantitative data relevant to the assessment of the work and benefits of the ECOC in each city;

-Interviews with managing teams: those responsible for the direct operational design and management of the ECOC projects were interviewed both during 2016 and again in 2017. Almost all of the key positions linked to the delivery agencies were involved in the interviews, including those linked to strategic development, marketing and communication, project implementation and financial management;

-Interviews with key stakeholders 13 : interviews were undertaken with stakeholders both directly or indirectly involved in the planning, management or delivery of the ECOC along with those more widely linked to the cultural, social, economic or political agenda of the host cities. Stakeholders included personnel working in cultural organizations, in city/region/State level administrations, in tourism and visitor agencies, in media organizations as well as in voluntary and community organizations;

-Surveys of ECOC projects: a specific survey of projects was undertaken by the evaluators in Wroclaw, whilst in Donostia San Sebastian a survey already undertaken by the Donostia San Sebastian Foundation (the body responsible for the ECOC delivery) was used to gain further insight of project managers' views on a variety of different issues linked to the design, delivery, benefits and legacy of the ECOC 14 ;

-The evaluation does not include a wider public consultation. As explained in the roadmap 15 , the Action is considered to be local. International participation is scattered within and outside Europe and is difficult to reach. On the other hand the opinions would be based on attendance to specific events and would not give useful insights for the evaluation of the ECOC Action as a whole.

The final report of the contract supporting the evaluation provides a detailed understanding of the 2016 ECOC Action and within this an assessment of the work and progress of Donostia San Sebastian and Wroclaw. There are however issues to consider when assessing the strengths of the evidence base used for this study:

-Although both cities have undertaken some form of evaluation work themselves, part of the results of these studies were not yet available at the time of the evaluation. The European evaluation of the ECOC Action has used as much of this secondary information as possible, but could not benefit from its final results, especially for what concerns quantitative data;

-As already mentioned in previous evaluations, an ideal way of conducting this evaluation would entail a before ('baseline') study and an after-picture ('ex-post') study, instead of carrying out the latter alone; the impossibility of comparing the two studies affects the accuracy of the evaluation. Moreover, since the evaluation was undertaken during and shortly after the end of the ECOC time framework some of the effects of the programme had not manifested themselves entirely. Many stakeholders involved in the evaluation commented that the real impact of the ECOC on the city and its residents would take time to filter through. In this respect, the Commission once again highlights that budget 16 and timing 17 only allow an ex-post evaluation to take place and therefore only an after picture has been studied; 

-An ulterior consequence of the modest budget allocated is the fact that the primary evidence data gathering tends to be more of qualitative than quantitative nature; while qualitative data still holds a great importance in the evaluation, lack of quantitative data translates into a lesser dependability, for instance, in the process of proving the objective outcomes and impacts of ECOC on widening participation in culture.

5.Analysis and answers to the evaluation questions

This part of the report presents a diagnosis of the implementation of the ECOC Action in 2016. It draws together the results relating to the two ECOC 2016 and to a lesser extent the findings from previous from ECOC evaluations.

All figures appearing in the following sub-sections have been directly drawn from the report of the external ex-post evaluation of the two ECOC 2016. The reader will find in the full text of the document 18 many more examples illustrating the conclusions presented below.

Very often, examples and figures can illustrate different points made under "relevance", "efficiency", "effectiveness", "sustainability", "EU added value" and "coherence", and are therefore sometimes repeated in various sub-sections. However, for the ease of the reading, repetitions have been limited to the minimum and the reader is invited to correlate some conclusions presented below with examples or figures that may have been given in previous paragraphs or may be given in following sub-sections.

5.1.Relevance

Findings from the 2016 ECOC evaluation show that the two title-holders have developed and implemented a cultural programme that is consistent with the EU Treaty and in particular Article 167 19 . Indeed, in 2016 the ECOC Action continued to help EU Member States promote cultural diversity as well as highlight common elements, and added to the "flowering" of their – and European – cultural scene. Indeed, as will be shown in following sub-sections, in the two cities, thousands of events took place on top of the usual cultural offer, covering all types of cultural and artistic disciplines and presenting various aspects of the European cultural diversity notably through co-operations with partners from other (European) countries.

Although both cities clearly understood the strategic objectives of the ECOC, the programme in Wroclaw was more relevant to the EU Action objectives compared to that of Donostia San Sebastian.

Donostia San Sebastian encapsulated well enough a European dimension at project level, with its three key themes, namely multilingualism (with projects on European minority languages, such as Basque, Welsh, Cornish and Gaelic), cohabitation (with projects on European conflicts and how to avoid and overcome violence), and multiculturalism and migration (with projects focused on human rights, the roles of women or the coexistence with several religious identities often being found in one place). According to the survey of participants, 30% of the projects specifically promoted a sense of "European belonging" with a further 24% stating that they collaborated with European partners whilst developing and delivering their projects.

The project "Europa Transit" is emblematic in this sense; this travelling "embassy" visited ten cities that had lived through wars and conflicts: Ceuta (Spain), Belfast (UK), Dresden (Germany), Wroclaw (Poland), Sarajevo (Bosnia-Herzegovina), Moscow (Russia), Paphos (Cyprus), Plovdiv (Bulgaria), Pristina (Kosovo) and Thessaloniki (Greece). The work acted as a spotlight to share how people overcome conflicts and find mutual understanding with people who have lived through a similar history. The project also included a "pan-European Mashup" which presented stories, pictures and videos from the locals of the places visited on topics such as discrimination, injustice, solidarity, dignity and fear.

Another project worth mentioning in this respect is 'Peace Treaty'. With an international aspect, it included a series of exhibitions, contemporary artistic productions, publications, seminars and conferences on the representation of peace in the history of art, culture and law. In this way, it used peace as the main theme and used culture to communicate moments of Western Society that included peace treaties that configured to Basque Society. Part of this looked at reasons why peace is important, the reasons why peace does not exist and the challenges of not having peace.

However, as highlighted by a majority of the stakeholders taking part in the European ECOC evaluation, the global objective of Donostia San Sebastian 2016 was more focussed on local issues and priorities – through a relatively large number of smaller rather intimate cultural activities found throughout the year instead of a range of high profile and 'big hitting' events such as large concerts, exhibitions and festivals. This was very much linked to the needs and context of the city (i.e. dealing with its recent past of violence and reconciling its communities) and the strength and international connections of its existing cultural offer prior to the ECOC year (and the correlated absence of need to seek for much improvement in this respect). In this sense, it shows that all ECOC will also reflect local circumstances and ambitions, despite the ECOC being an EU title.

Donostia San Sebastian focussed on using the ECOC to strengthen the cultural software rather than hardware of the city, the latter being already quite developed in the city prior to the ECOC year. Rather than developing new buildings or facilities like Wroclaw did, the city used the ECOC to put on a sensitive cultural programme that partly helped the citizens of the city overcome some of the difficulties of the past terrorist activities of ETA – thus meaning it was very relevant and unique to the city and its past. 

On the other hand, it appears from the final report of the contract supporting the evaluation that Wroclaw used the ECOC to further strengthen the European character of the city, raise its international profile and, crucially, explore some of the more difficult elements of its recent history. It also implemented the ECOC as part of a wider strategy to develop the city on the basis of culture, tourism and sport.

Indeed, several themes of European significance were presented and explored by the cultural programme. These were diverse and included issues such as Polish-German reconciliation, cultures from different European countries and review of trends emerging in the work of young artists from across Europe.

The majority of respondents to the survey of organizations that implemented projects reported that their projects had a European dimension: 50% said that their projects sought to increase awareness of cultural diversity, 50% that their projects highlighted intercultural dialogue and mutual understanding and 45% that their projects included themes of relevance or significance across Europe with only 30% reported no European dimension.

In this respect, "Migrations and Metamorphoses" was a particularly important theme in the promotion of the ECOC European dimension; it was dedicated to the cultural effects of the population transfer that took place in the city at the end of World War II 20 , but also on the modern phenomenon of migration into and across Europe. Among the many relevant projects to this theme there was "Unfinished Palace. About people, migration, and borders", that consisted in a series of multimedia events at the former Świebodzki Railway Station over two weeks, including exhibitions, concerts, readings and performances, as well as "Open Call Europe", a long-term and interdisciplinary artist-in-residence programme which invited artists, art curators, cultural activists and managers from the 27 EU Member States, as well as Albania, Bosnia and Herzegovina, Georgia, Iceland, Israel, Kosovo, Liechtenstein, FYROM, Moldova, Montenegro, Morocco, Norway, Serbia, Switzerland, Turkey and Ukraine.

Wroclaw 2016 also featured numerous collaborations with cultural operators from other European countries and elsewhere in the world. For example, the "European Student Symphony Orchestra", which created an orchestra with young musicians from selected European schools, which then performed in Wroclaw and elsewhere in Poland and internationally.

Furthermore, with the view to connecting with audiences from abroad and further profile Wroclaw in other cities, events were also organised in other countries, such as a major exhibition on "The Wild West. A History of Wroclaw's Avant-Garde" in Warsaw, as well as in Bochum (Germany), Budapest (Hungary), Kosice (Slovakia) and Zagreb (Croatia).

Of those organizations responding to the project survey, 81% reported that their project involved cultural organisations or artists from other countries in some way (presence of performers from other countries, exhibition of works from other countries, international cultural exchanges, performance of Polish performers abroad or collaboration with non-cultural organisations or people from abroad). As a whole, projects reported collaborations with organisations or artists from 42 other countries, mostly in Europe.

5.2.Efficiency

Overall, and despite the absence of a strong costs-benefits analysis, the report concludes that the ECOC remains an efficient EU Action, providing good levels of benefits for the EU for relatively little EU investment. Indeed, the only direct contribution from the European Union, in the form of the €1,5m Melina Mercouri prize awarded to each ECOC, is dwarfed by the total amount of money invested by the host cities in designing and delivering the ECOC (the operational budgets were approximately €86,4m for Wroclaw and €49,6m for Donostia San Sebastian).

Without the initial EU impetus to support the ECOC then it is unlikely that the host cities would have invested anywhere near the amount of funding they did in connection with the ECOC title. The possibility of securing the title typically stimulates cities – but also their respective regional and national public authorities – to invest much more heavily in their cultural offer than they would in the absence of the ECOC – both in terms of infrastructure (this was more the case in Wroclaw than in Donostia San Sebastian) and expenditure on cultural events and operations. The independent evaluation concludes that this means the Action remains highly efficient in terms of returns from the Melina Mercouri prize.

On the other hand, the final report of the contract supporting the evaluation also highlights that delivering an ECOC is still very challenging at the city level from an efficiency point of view. Any city considering applying for ECOC status needs to recognise that it will need to generate significant levels of funding from various public and private stakeholders if it wants to deliver a strong cultural programme and that the returns on the investment are sometimes intangible (e.g. better access to culture).

In this respect, it is worth mentioning that Donostia San Sebastian had difficulties in securing the budget initially sought for. Ultimately, it had to deliver a programme with a budget that was 44% lower than in its original planning. The Foundation itself was relatively poorly resourced in terms of staff before 2016. Just three years before the ECOC year it had less than 5 staff members and by 2014 staff numbers were still relatively modest (13). Despite this, looking at the visitors' numbers, the audience figures (1,08 million, despite the quasi-absence of 'big ticket' events, which are expensive to produce) and the level of cultural activity (3,475) it is clear that high levels of efficiency were achieved. As far as Wroclaw 2016 is concerned, it was on the contrary able to attract a very high level of financial contribution from the State (€31.7m, representing 37% of the whole operational budget) as well as of corporate sponsorship in culture (€7m).

Like in previous years, the ECOC in 2016 has stimulated many thousands of 'extra' cultural activities and the cities have both gone out their way to generate much more depth and volume of cultural activity compared to their usual cultural offer – approximately 2,000 more activities than the standard figure in Donostia San Sebastian; and 2,000 artistic events as a whole in Wroclaw, many of them new and in new settings such as urban spaces (see following sub-section). New activities, new resources, new audiences and new cultural infrastructure (as illustrated in following sub-sections) have all been generated across 2016 as a direct consequence of the ECOC.

The delivery mechanisms established in both San Sebastian and Wroclaw were strong and successful. Donostia San Sebastian delivered its programme in relatively difficult circumstances including political changes at various levels of territorial governance (City in 2011 and 2015, but also Province, Region and State in 2015 and 2016) and comparatively high levels of political influence, making 'culture to overcome violence' even more of a 'brave' subject to engage with, when it would have been easier to deliver 'more of the same' type of culture. This higher than normal (in comparison with other ECOC) political involvement brought about a range of issues that were often seen as adding an extra level of complexity to the governance structures of the ECOC, in particular because it meant that some key decisions (for example, on what the legacy plans of the programme should be) were delayed or sometimes not taken at all.

The governance arrangements in Wroclaw were generally stable throughout the development and implementation stages, albeit with a very important change in the artistic direction (i.e. the move from a single Artistic Director to a Board of Curators). The consensus amongst stakeholders was also that Impart (the delivery body) and the Curators enjoyed a high degree of artistic independence and freedom from political control at local and national level (within the usual constraints associated with accountability for public expenditure) and operated efficiently.

5.3.Effectiveness

Ultimately, the effectiveness of the ECOC in achieving the objectives set for them (at EU level and local level) can only be determined in the long-run. That being said, the final report of the contract supporting the evaluation however shows that both cities have proved successful in fulfilling the short-term objectives, most notably the implementation of extensive, innovative cultural programmes, which had a European dimension and involved local citizens (both title-holders) and international visitors (Wroclaw more than Donostia San Sebastian). It appears from the final report of the contract supporting the evaluation that none of the stakeholders in either of the two cities felt that more could have been done and that 2016 was ineffective, a waste of resources or should have been done significantly differently. The final report also mentioned that the overriding 'feeling' on the effectiveness of their ECOC was one of positivity and supportiveness towards their title year, even for those stakeholders who were interviewed specifically because they were initially against the ECOC.

Effectiveness in delivering a cultural programme with a European dimension

Donostia San Sebastian succeeded in delivering a cultural programme that was very wide, varied and more extensive than in in a 'non-ECOC' year, for an expenditure of almost €28m (62% of the total expenditure of the ECOC year). The programme consisted of 3,475 activities (as against 1,500 in a normal year), with theatrical activities making up 33% of the programme, musical activities 25%, audio visual 20%, literacy 7% and the remainder being a mix of other forms of art (i.e. visual arts etc.). Those designing the original cultural programme were keen to not simply put on 'more of the same' in relation to delivering or repeating more 'film festivals, jazz and high-quality art exhibitions' that the city already had and which local people were used to. According to the local evaluation that was carried out, 70% of the projects were specifically put on for 2016 and did not exist prior to the year. Furthermore, 65% of the projects said that they would not have gone ahead without the stimuli (the funding) of the ECOC.

The Donostia San Sebastian's programme didn't just amplify the city's cultural offer, but it clearly differentiated it from the pre-existing cultural activity, using culture as a vehicle for healing wounds inherited from the past.

However, although the ECOC in San Sebastian included themes and projects which had a European dimension some of the strategic stakeholders taking part in the evaluation felt that the overall programme generally had a firm local rather than European focus. Despite some of the topics being relevant at the European level, like terrorism, cohabitation and multilingualism – which were strongly present in the programme, some stakeholders felt that the ECOC targeted and supported local citizens and culture rather than European ones. This has to be read keeping in mind that the content of the cultural offer of the city before and outside of the ECOC was already very European and to a large extent already had aspects with a strong European dimension – for instance, the San Sebastian International Film Festival and the San Sebastian Jazz Festival.

In Wroclaw, the ECOC cultural programme was more extensive, innovative and European in nature compared to the city's cultural "baseline" offering in previous years. It should be noted that the city's cultural offering had been steadily expanding in scope and diversity over the ten or more years preceding 2016, reflecting the long-term strategy of the City of Wroclaw. However, the title-year saw a further, marked increase in both the scale and the diversity. As a whole, 424 projects totaling around 2,000 artistic events took place. They were articulated in the eight disciplinary sub-programmes indicated above and represented an expenditure of almost €27m (47.4% of the total expenditure of the ECOC year). Of the organizations that responded to the project survey on cultural capacity, 30% reported that they implemented entirely new activities, 39% said that their activities were operated at greater scale in 2016 while 30% reported that their activities were undertaken at the same scale in 2016 as in previous years.

The eight curators of the eight disciplinary sub-programmes were provided with the opportunity and the resources to test new ideas. Some new activities with a European dimension were introduced into Wroclaw's cultural offering, including a programme of artistic residences for artists from other European countries. Of the organisations that responded to the project survey, 72% believed that the Wroclaw2016 cultural programme featured a good balance of traditional and avant-garde cultural expressions and 80% believed that it was of high artistic quality. Furthermore, among the projects responding to the survey, 81% had involved cultural organisations or artists from other countries. Just to give a few examples, Wroclaw hosted the "Theatre Olympics", an international theatre festival, which presented the achievements of some of the most prominent theatre practitioners from around the world, and its New Horizons International Festival was extended to include masterclasses as well as a presentation of Basque cinema as a way to showcase the other ECOC 2016.

Effectiveness in reaching and engaging with local citizens and in targeting specific groups

As already mentioned, Donostia San Sebastian used its ECOC to tackle a complex and sensitive issue, namely past violence and the differences that the city's communities have with one another, and the focus was perhaps more than in any other ECOC on local citizens. It is difficult to understand the effectiveness that the ECOC had on this central issue in a quantitative or 'empirical' way. Despite this, the report supporting the evaluation concludes that the ECOC went hand-in-hand with a very solid and substantial local involvement and was effective in helping people open up about their differences and also talk about the past issues that faced the city.

According to the local evaluation, 60% of the projects involved local people in some way (undertaking research with local residents and organizations, undertaking in-depth work and co-designing cultural activities with them or involving them in the actual delivery of cultural activities). The project "Forum Theatre: what about you" is a perfect illustration of how citizens were pro-actively involved. This project addressed feelings, contradictions and archetypes provoked by violence and social unrest. It used the model of the "Theatre of the Oppressed", giving the audience the power to decide the development of the narration. 40 sessions of this project successfully took place with good audience throughout the year. Moreover, a volunteer programme run centrally by the Foundation counted up to 2,373 volunteers in 2016 (with 41% of them not having any volunteering experience before the ECOC year), amounting to 10,493 hours of volunteer time. Citizens' participation was further stimulated by the "Waves of energy" project, which gave local individuals and non-profit organizations opportunities to present their cultural projects for development, of which sixty-six were financed with 60 grants of €2,000 and six of €20,000. Some of these projects were really intimate and personal, and were meant to help people deal with the violence they had experienced in their own lives. Although staff in the Foundation was involved in facilitating the project, local people made all the key decisions in terms of who and what themes benefitted from the funding.

Against this backdrop, there is little evidence of the ECOC in Donostia San Sebastian targeting groups in the city with specific activities of the cultural programme – for instance, designing projects specifically for older residents or the disabled. Having said this, audience figures show that 58% of the audience of ECOC projects were female and that 15% were under 25 and nearly half were under 39 years old. More generally, the baseline survey undertaken by the local evaluators shows that the number of local people that 'actively participated in culture on a regular basis' (twice a month) grew slightly from 60% in 2015 (a high figure already) to 64% in 2016. However, the report also highlights that the absence of 'mega events' negatively impacted on local people attending ECOC projects.

As far as Wroclaw 2016 is concerned, it also involved a large numbers of citizens, including children and young people: while around 2,000 people served as volunteers, numerous projects and events enabled citizens to participate as creators and performers. In total, more than 170,000 people actively participated in projects, of which 50,000 were children and young people (e.g. 15,000 local school pupils participating in the educational programme related to the Film sub-programme with the support of the educational and academic community). 

For instance, the "Backyards" project sought to initiate a creative process based on collaboration between artists (most coming from outside Wroclaw), local residents, property owners and other local institutions. The artists were invited to select one of the many backyards in the city and generated ideas for a creative activity. Once the backyard had been selected, the project co-ordinators would then identify the owners of the backyard and thus seek the relevant permissions to operate. The artists would then begin the fieldwork and the consultation with residents. Over a two-year period (2015-16), a total of 32 projects took place in 42 backyards, 7 estates and 5 backyard walks. The artistic activities varied and included lighting installations, workshops for children, exhibition of residents’ objects, collecting residents’ stories and making them available in audio form, involving residents in one block writing letters to residents in another block etc.

Contrary to what happened in Donostia San Sebastian, many of the activities to encourage the participation of citizens were aimed at specific groups. The respondents to the survey of organizations that implemented projects particularly highlighted children (75% of projects), older people (70%), people with disabilities (50%), poor or disadvantaged communities (40%) and minority ethnic groups (30%). As an example, the intention behind the music project "Yiddish and Ladino" aimed at engaging the many societies and artistic groups associated with ethnic groups living in Wroclaw, in this case by opening up the city for a presentation of the Jewish culture. Another example was the "Adapter – Cinema Without Barriers" project, targeting at people with hearing and visual impairments, which included educational activities and a video-on-demand portal of more than 100 film with audio-description and subtitles. There was also a special focus on involving foreigners, for example through the project "Polish Cinema for Beginners", a series of screenings with English subtitles targeted at foreigners living in Wroclaw.

The most prominent project in this aspect was "MikroGRANTY" ("micro-grants"). To effectively build local cultural capacity and facilitate the participation of local groups and citizens, this programme was open to applications from private individuals, informal groups, NGOs, home-owner associations, social cooperatives, church institutions, religious organizations and other similar groups. Applicants could submit projects relating to any artistic discipline, but their artistic expressions had to relate to the overall ECOC concept of "Spaces for Beauty", chosen slogan for the ECOC programme whose intention was "to create spaces within which to restore the presence of beauty in public life and in daily habits". In total, 115 projects were supported, with a grant support of up to about €1,175. Taking only those projects implemented in 2016, some 190 partner bodies were involved, as well as 319 people (including 179 volunteers) and 3,690 active participants were reached with a total of 12,000 "likes" on Facebook.

Effectiveness in reaching out to European and international audiences

The final report of the contract supporting the evaluation highlights that the local orientation of the ECOC in Donostia San Sebastian made fruition of the ECOC activities more difficult for non-locals. As an example, although some of the literature associated with the ECOC was found in different languages (e.g. the monthly 'What's on' publication was 40% in Basque, 40% in Spanish, 10% in English and 10% in French), the proportion of literature translated into English was relatively low compared to other ECOC.

Against this backdrop, it however appears from the report that 7.6% of the ECOC audiences were from another European country – this is in line with the average ECOC level, which is estimated between 5-10% – and that out of the 1.08 million people who attended ECOC events, 84% came from Donostia San Sebastian or its close surroundings. Furthermore, figures from the tourist agency show that the city saw a slight uplift in foreign visitors by 1.5% in 2016 compared to 2015, but it is not possible to directly connect this increase with the ECOC.

In Wroclaw, one of the explicit goals of the ECOC was to increase tourism and attract international visitors. In spite of a very modest budget, Wroclaw run an effective marketing and communication campaign targeting among others Europe and the world, which put a considerable emphasis on co-operation with other agencies (for joint representation at promotional events and tourism fairs) as well as on attracting international journalists. In total, Impart organized visits by 153 journalists from 26 countries and welcomed about another 100 who organized their own visits.

Given Wroclaw's history and location, there was a particular focus on Germany. For example, with the co-operation of the relevant transport bodies, a special direct train service from Berlin was established: the "Culture Train", operating at the weekends from 30 April 2016 onwards. The "Culture Train" itself presented a range of culture, including literature, music, theatre and exhibitions and more than 40 artists participated in the programme. In total, nearly 23 000 passengers made use of the train in 2016, and given these passenger numbers, it was decided that the service would continue at least until the end of 2018.

According to research undertaken on behalf of the City of Wroclaw, around 5 million tourists visited Wroclaw in 2016, of which 1.6m came from other countries, marking a substantial increase in international tourist stays during the year. Data from the Central Statistical Office of Poland shows that 2016 was a year of growth in tourist stays in Wroclaw (+5,2% compared with 2015, after already two years of sharp increases) and that that an extra 50,000 international tourists stayed in the city's hotels in 2016 compared to 2015.

5.4.Sustainability

The two 2016 ECOC cities offer very different potential for the sustainability of their activities and of improved cultural governance.

For what concerns Donostia San Sebastian, the main finding of the evaluation is that legacy planning of the ECOC has been generally limited, with a lack of a clear and formal strategy from the outset: compared to other ECOC bid books, there was in Donostia San Sebastian's bidding document very little detail on the planned activities to sustain engagement with the themes and projects of the ECOC year. One driver behind this shortcoming was identified through consultations with stakeholders as the deficiency of joint understanding by the political parties on what an ECOC legacy should look like. The process of shaping this perspective fell under the responsibilities of the ECOC Board (rather than the Foundation).

Although the cultural operators were all generally still in place and still delivering strong cultural content beyond 2016, this was more due to the strength of the existing cultural infrastructure found in the city, rather than any specific legacy work from the ECOC programme. While there hasn't been an outstanding increase in cultural capacity – which was already extremely well developed in the city prior to the ECOC year in the city and still remains of an excellent standard – the ECOC year highly motivated stronger linkages between local players: almost 80% of organizations delivering an ECOC project worked in cooperation with other actors in order to deliver their projects and 67% of them were new relationships that had previously not existed prior to 2016.

Another aspect of the legacy issue in Donostia San Sebastian is linked to a lack of new cultural buildings: as stated earlier, the ECOC was generally focused on cultural content/events rather than buildings/infrastructures, making the lasting legacy of the programme even more difficult to grasp.

As in many ways the goals of the ECOC in San Sebastian, its results and focus were socially and politically oriented with a strong focus on reconciliation, its legacy is more of a "soft" nature, and articulating its impact is therefore harder. Even stakeholders taking part in the evaluation found it hard to express the legacy of a cultural theme aimed at 'peace' or community cohesion compared to describing the lasting legacy and impact of a new theatre or opera house. Moreover, unlike Wroclaw, Donostia San Sebastian did not have a strong ambition to improve tourism by linking it to the ECOC Action, thus making yet another indicator of objective success of the Action unavailable.

Despite the difficulties outlined above, a key result of the programme overall was often recognized by stakeholders taking part in the European evaluation as being one which brought the political parties closer together to actively work on a 'city level' project and on an activity that was genuinely 'positive', therefore influencing in the longer-term the way the city operates internally.

On the other hand, legacy planning was much developed and coordinated in Wroclaw as the ECOC was implemented as an integral part of a long-term development strategy with strong and linear political support at local level, and therefore offers greater potential for long-term sustainability of the ECOC impacts. This strategy is also being informed by the various research and evaluation activities undertaken by the University of Wroclaw, which included recommendations for enhancing the "mikroGRANTY" programme beyond 2016: for the first time, a dedicated funding stream will be available to nurture grass-roots cultural initiatives, not only through finance but also through the technical and administrative support offered by Impart.

On the long-term scheme, Impart together with the City's Departments for Social Affairs and for Culture have put forward a proposal for a strategy for 2017-2020 and beyond – "Culture – The Current!" ("Kultura – Obecna!") –, building among others on the experience of the ECOC. The strategy lists four strategic objectives (and 31 recommended actions):

-Maintaining the prestige of Wroclaw culture;

-Developing the cultural infrastructure of Wroclaw;

-Ensuring conditions for the development of culture; and

-Ensuring an increase in participation in culture in Wroclaw.

Additionally, Wroclaw has the potential to be sustained by the continued operation of Impart from its premises in the city centre and with many of the key staff members still in post. Impart will therefore continue to play a key role in coordinating cultural activities in the city, as well as in implementing specific projects. Furthermore, other major events such as the World Games 2017 (an international multi-sport event involving sports that are not contested in the Olympic Games) already built – or will build – on the capacity and profile generated by the ECOC.

As is often the case with ECOC, Wroclaw 2016 will also have soft legacies in terms of stronger skills for its cultural sector, stronger relationships (in particular at European level), a stronger and more vibrant cultural life (that is the opinion of 74% of those organisations that responded to the projects survey) and – more widely – a higher profile for culture in the city (according to 87% of firms in the cultural and creative industries surveyed by the University of Wroclaw).

Finally, the ECOC in Wroclaw was supported by and provided an impetus to investments in the cultural infrastructures of the city, for instance with the opening of the National Forum of Music concert venue. Various locations in the city were also used in new ways as venues for cultural events, such as rivers and riverbanks within the city centre. The (re)opening of several new (or refurbished) venues and the general modus operandi of taking culture out of the main venues in the city centre and "spread" it around many other, often new or unusual locations will also provide with a tangible kind of legacy. Reflecting this, 83% of respondents to the project survey reported that the ECOC had improved local cultural infrastructure.

5.5.Coherence

The ECOC has stimulated the two cities to use European Regional Development Fund (ERDF) for investments in cultural infrastructure and has given greater impetus to the completion of those investments in time for the title-year.

In particular in Wroclaw, there was a wider strategy for the development of culture and tourism from the early 2010s that foresaw investments in physical infrastructure and facilities.

It included improvements in the rail, road and air transport infrastructure of Wroclaw, some of which was funded by the ERDF. Funding from the ERDF also supported the creation or renovation of cultural infrastructures in the city – for instance, nearly €41m of ERDF funding was invested in the new venue of the National Forum for Music which opened in September 2015. In turn, the ECOC helped raise awareness of this new cultural facility and contributed to the NFM's considerable success in attracting audiences in its first full year of operation (more than 500,000) as much of the music programme of the ECOC took place at the new NFM with for example a series of 14 concerts exploring "1000 Years of Music in Wroclaw" all along the year.

EU cross-programme funding has also helped maximize the results of the ECOC Action. For instance, Donostia San Sebastian's project "With Corners of Europe", that connected six Basque artists with another 60 European ones, was co-funded by the EU's Creative Europe programme and produced activities in Umea and Stockholm (Sweden), Ljubljana (Slovenia), Prizren (Kosovo), Gdansk (Poland), Zagreb and Rijeka (Croatia), Donostia San Sebastian (Spain), as well as Belfast, Northumberland and Middlesbrough (UK).

The ECOC Action is therefore relevant and complementary to a variety of EU policies and programmes, impacting not just cultural stakeholders but also those related to employment (with its impact on capacity building for example), enterprise (as an instance, 14% of firms from the cultural and creatives sectors in Wroclaw were involved in the ECOC, i.e. about 450 out of +3,000 firms, while 52% of them felt that they had derived commercial benefits from the ECOC and 40.7% reported an increase in turnover during 2016) or tourism as illustrated in the various sub-sections above.

5.6.EU added value

As already mentioned and illustrated above, the ECOC Action has achieved an impact that would not have arisen through the actions of Member States alone.

The designation of Donostia San Sebastian and Wroclaw as ECOC has attracted benefits that would have been unlikely to arise to the same extent in the absence of the Action. The 'label' itself is a key aspect of the EU added value of this Action as it acts as a significant generator of interest from stakeholders not only from the city and but also from far beyond; for example, Wroclaw's growth in terms of corporate sponsorship, cultural participation, visibility and local/international tourism was fairly consistent with and directly connected to the fact that the city was European Capital of Culture in 2016.

Moreover, for both cities there was great scope for European cooperation in terms of partnership and transfer of good practices, both from one ECOC to the other and in connection with other European organizations.

The results consistently achieved by the Action through the years establish that its models and mechanisms – for example in terms of building a solid governance for the delivery of the ECOC, increasing the capacity of local cultural organizations or attracting projects' ideas from local residents – can be applied not only regionally or within an individual Member State but also EU-widely, thus finally confirming its EU added value.

6.Conclusions

The lack of baseline data to be integrated in a comparative study of the city prior to the win of the title, start of title year and after the implementation of the ECOC year is a limitation to the study carried out by the contractors. These data are considered of paramount importance in order to achieve a balanced perspective, supported by a cogent and ample data basis, of the actual impact of the Action on a city. However the budget allocated to the evaluation work (approximately 70 000 € each year) is proportionate to the low level of EU funding directly provided to the ECOC (€1.5m Melina Mercouri Prize) and doesn't make it possible to have a before ('baseline') study and an after-picture ('ex-post') study. An ulterior consequence of the modest budget is that the primary evidence data gathering tends to be more of qualitative than quantitative nature; while qualitative data still holds a great importance in the evaluation, lack of quantitative data translates into a lesser dependability, for instance, in the process of proving the objective outcomes and impacts of ECOC on widening participation.

The following conclusions are therefore substantiated by an ample basis of qualitative and "soft" data (e.g. the views and opinions of various types of stakeholders) more than by a comprehensive quantitative set of data. The Commission is fully aware of – and accepts – those limitations.

The provisions of Decision 445/2014/EU, which will cover the ECOC titles from 2020 to 2033, foresee that the cities (starting with the ECOC 2020) shall be responsible for the evaluation of the results. The requirement to have a monitoring and evaluation plan is therefore part of the criteria used to assess the applications from bidding cities and should allow for the collection of data at early stages so as to establish baselines. This requirement is already an element of the monitoring process cities designated for the ECOC titles from 2020. As specified in Article 14 of Decision 445/2014/EU, the Melina Mercouri Prize money will be paid by the end of March of the year of the title, provided that the designated city concerned continues to honour the commitments it made at the application stage, complies with the criteria and takes into account the recommendations contained in the selection and monitoring reports. To give evidence that the commitments they made at selection phase are honoured, cities have among others to show that "the plans for the monitoring and evaluation of the impact of the title on the designated city are in place" at the end of the monitoring phase.

On top of that, according to Article 16 of Decision 445/2014/EU, the Commission will ensure external and independent evaluations that "shall focus on placing all past European Capitals of Culture in a European context, allowing comparisons to be drawn and useful lessons to be learned for future European Capitals of Culture, as well as for all European cities". Those evaluations will "include an assessment of the action as a whole, including the efficiency of the processes involved in running it, its impact and ways in which it could be improved." The first of these evaluations is due in late 2024.

Against this backdrop, the Commission concludes that the ECOC Action is highly valued by the hosting cities that can obtain positive impacts during the year, as well as during the preparation phase. The action also remains relevant at EU level. However, the long-term impacts of the two ECOC 2016 cannot be assessed at this stage as it is too early after the implementation of the ECOC year.

The Commission also concludes that the programmes implemented by the two 2016 title-holders were innovative and consistent with the objectives of the ECOC Action; they reflected its European dimension (in particular Wroclaw, to a lesser extent Donostia San Sebastian), involved local residents and stakeholders (in San Sebastian with 60% of projects involved local people in some way according to the local evaluation and 10,493 hours of volunteer time while Wroclaw designed cultural activities aimed at specific groups), brought culture to new audiences through specific strategies (such as "Waves of Energy in Donostia San Sebastian and "MikroGRANTY" in Wroclaw) and may lead to some legacies both physical (in the form of new or refurbished cultural and logistic infrastructures, as it happened in Wroclaw) or intangible (by creating a space and vehicle that helped communities talk about, understand and come to terms with past differences and helping the city and its residents coexist with one another in the future, as it happened in Donostia San Sebastian) although a proper legacy planning is lacking in Donostia San Sebastian.

Annex 1: Procedural information

1.Lead DG, Decide Planning/CWP references

The evaluation was led by Directorate General for Education, Youth, Sport and Culture (DG EAC). It is included in the Work Programme of Creative Europe for 2016 and in the Agenda Planning with the reference 2016/EAC/026.

2.Organisation and timing

The evaluation was supported by an external and independent evaluator, under a service contract. The service contract was implemented via a Framework Contract with reopening of competition and in accordance to the Financial Rules Applicable to the General Budget of the Union 21 and its Rules of Application 22 .

The evaluation Roadmap was adopted on 13 June 2016 23 .

According to the Roadmap, a Steering Committee including staff from DG EAC and from the Secretariat General was established in April 2016. The Steering Committee met in four occasions: to prepare the Terms of Reference (ultimately sent to contractors in July 2016), to approve the Inception Report in January 2017, to discuss the draft final report in June 2017 and to approve the final report in November 2017. Extensive correspondence between the Steering Committee members was held in between the meetings to follow-up on the evaluation.

The evaluation initial schedule foresaw a final report in the third quarter of 2017. Due to the late availability of data used for the evaluation (and gathered by the cities) it was agreed to delay the submission of the final report to the fourth quarter of 2017.

The evaluation did not need to be submitted to the Regulatory Scrutiny Board, as it is not considered a major evaluation, does not contain an impact assessment and does not constitute a fitness check for legislation.

3.Exception to the better regulation guidelines

The only exception is that the evaluation does not include a wider public consultation. As explained in the Roadmap, the ECOC Action is considered to be local. International participation is scattered within and outside Europe and is difficult to reach. On the other hand the opinions would be based on attendance to specific events and would not give useful insights for the evaluation of the ECOC Action as a whole.

4.Evidence, sources and quality

The 2016 evaluation of the ECOC used a series of data sets to inform its findings. The main ones being:

-Interviews with over 59 stakeholders directly and indirectly involved in the planning, operation and delivery of the two ECOC programmes;

-Two surveys of projects that were part of the two ECOC programmes who delivered the cultural programme attached to the intervention;

-A literature review of ECOC and European level information on the two ECOC including application/ bid information, EC Committee reports, cultural programme brochures, web sites and news articles. The ECOC's own external evaluations have also been used to inform the European evaluation process.

-Together, the above evidence base provides the evaluation with a valid and rounded set of data to inform the views on the main aspects of the ECOC evaluation including efficiency, effectiveness, sustainability and relevance. This view has been informed by:

-The scale of the consultation exercise. Over 150 individuals have fed their views and opinions into the evaluation process either through face to face interviews, telephone interviews or through the project level survey.;

-The nature of the consultation exercise. The evaluators were keen to consult with those who had a more indirect and external view of the two ECOC. These stakeholders including journalists, those not directly benefitting from the ECOC (e.g. rejected projects) as well as those working in the wider cultural policy agenda at regional and city level. This ensures the evaluation is not simply based on those who benefitted the most from the ECOC.

Annex 2: Stakeholder consultation

The stakeholders were consulted via targeted consultations (interviews or phone interviews) and via an on-line survey, mainly on projects that had participated in the ECOC year or had submitted a proposal that was rejected. For Donostia San Sebastian, the on-line survey of projects (survey questions below) was implemented by the contractor supporting the evaluation, while for Wroclaw the contractor relied on the survey of projects that was implemented Impart, the ECOC implementing body itself and which questions aligned with the information sought.

The consultations included the team responsible for the implementation of the ECOC, the political statements involved in the project, the projects participating in the programme or having submitted proposals to participate that were rejected, and personalities attached to the cultural tissue in both cities. The objective of the consultations was to have evidence supporting the findings and conclusions of the evaluation. The questions used for the survey of projects in Donostia San Sebastian can be found in the report produced by the experts assisting the Commission:

- https://ec.europa.eu/programmes/creative-europe/sites/creative-europe/files/files/ecoc-2016-evaluation-en_0.pdf

For the reasons already mentioned in Chapter 4 of the Staff Working Document and in Annex 1, no open public consultation was conducted in the framework of the evaluation exercise. The consultation was restricted to relevant stakeholders in the two cities hosting the title. The characteristics of the Action and the scope of the evaluation do not make it necessary to extend the consultation to a wider public (as indicated in the roadmap published).

Annex 3 details the evaluation questions for which the stakeholders' consultation was used. These regard mainly the questions which answers are not based on factual data. The stakeholders' consultation was particularly useful to find information about the impact of the ECOC in the cultural offer of the city, the participation of citizens and local cultural operators, the building of capacity for local cultural operators and legacy prospects.

Annex 3: Methods and analytical models

The figure below presents the overview of the methodology. A more detailed overview of the methodology and sources used for each of the evaluation questions is presented in the subsequent tables and text.

1. Overview of methodology and tasks of the evaluation:

Table 1.1 Evaluation questions: Relevance, EU added value and coherence

Table 1.3 Evaluation questions: Effectiveness

Table 1.4 Evaluation Questions: Sustainability

2.Details of the relevant evaluation tasks and methodologies:

Inception phase

The aim of the inception phase was to finalise the evaluation framework and research tools, address the project steering group's comments regarding the proposed method and collect background information on Donostia San Sebastian and Wroclaw.

Initial consultations

The task consisted in identifying stakeholders to consult with, targeting people that are knowledgeable about the two cities, their cultural programmes and the achievements, lessons learnt and impact of the ECOC, including national, regional and local stakeholders. Discussion at the kick off meeting identified a need for strong triangulation and to gain opinions from a broader set of stakeholders.

During the initial first visit to the cities described below, the contractors supporting the evaluation also sourced stakeholders for consultation who did not directly benefit or take part in the ECOC programme and who they felt would have an honest, impartial and unbiased view of how the programme was developed or implemented. While the aim was not to look for purely negative views, the contractors wanted to speak with stakeholder who, for example:

-did not deliver an ECOC project or take part in the cultural programme attached to the ECOC;

-were a member of the local and national press who had written either an honest or negative piece on the ECOC (as opposed to simply writing a press release which, for example, communicated the content and dates of the cultural programme);

-had applied for ECOC funding (to be part of the ECOC cultural programme) and were rejected;

-were involved in ECOC activity but were often vocal about a particular aspect (e.g. the quality of culture on offer, the lack of audiences, the lack of diversity of cultural content);

-were not directly involved in the cultural policy agenda in the cities but instead had an economic, social or environmental view on the ECOC.

The list of stakeholders consulted is provided in the annex to the Report 24 . This list was partly developed by the ECOC delivery teams but also through an internet search for stakeholders who work in the above fields. The list was further expanded during the second visits themselves (through a snowballing technique - i.e. asking stakeholders for other stakeholders who could be useful "external" individuals to consult).

With this in mind, in Donostia San Sebastian, the list of stakeholders consulted includes:

·A journalist for the local newspaper who had written independent reviews of various cultural projects as well as the overall delivery of the ECOC programme;

·A cultural operator that applied for ECOC funding (to be part of the ECOC cultural programme) but was rejected;

·A cultural operator that was involved in the cultural scene of the city but did not directly benefit from the ECOC programme and also who were vocal about a particular aspect; and

·Local stakeholders not directly involved in the cultural policy agenda in the cities but instead had an economic, social or environmental view on the ECOC.

In Wroclaw, it included:

·An external evaluator from local university;

·A former Minister of Culture and National Heritage;

·An independent radio journalist;

·A television journalist; and

·The Head of Polish-German Chamber of Commerce.

First visits to the cities

Initial face to face visits were made, to Donostia San Sebastian on 28 November 2016 and Wroclaw on the 7 to 9 December 2016.

Those consulted with during the initial visit to Donostia San Sebastian were:

·Pablo Berástegui (Managing Director of the DSS2016);

·Xabier Paya (Cultural director of DSS2016);

·Garbiñe Muñoa (Financial Director of DSS2016);

·Enara García DSS2016 (Director of participation and evaluation of DSS2016); and

·Fernando Alvarez (Communication Manager of DSS2016).

Those consulted in Wroclaw were:

·Dominika Kawalerowicz (Head of Programme Team, Wroclaw2016);

·Katarzyna Mlynczak-Sachs (International Relations Manager, Wroclaw2016);

·Olga Nowakowska (Chief Financial Officer, Wroclaw2016);

·Jaroslaw Fret (Theatre Curator, Wroclaw2016 and Director of the Grotowski Institute); and

·Katarzyna Kajdanek (Associate Professor, University of Wroclaw, evaluator).

Desk research including big data feasibility

This involved collection and review of literature, primary data from the delivery agency and available secondary data, as well as web and social media statistics. The list of the literature that was collected at the European level which was reviewed and assessed for usefulness for the sake of the evaluation can be found in the Report 25 . Much of the European level literature is helpful to either understand the policy drivers at EU level which support the two 2016 ECOC or provide background on the bidding and application stages for both 2016 cities (i.e. little of it provides information useful for the content, delivery and impact of the two programmes). More academic literature at the EU level linked to the ECOC was also searched for. Instead of looking at academic literature generally on 'European culture' and also literature before 2016, the contractor supporting the evaluation specifically looked at up to date (i.e. produced in 2016) pieces which linked to the ECOC Action. This search did not prove fruitful, although some academic literature at the city level was found.

To complement the above desk research task the contractor assessed the extent to which each ECOC had used big data as well as analysed web and social media to increase visibility and interest in the ECOC among country residents and internationally.

Online survey of projects

The survey of projects proved to be an important source of complementary quantitative data for previous evaluations, and was done for 2016, supported by additional qualitative interviews of projects. The survey was performed by the contractor supporting the evaluation for Donostia San Sebastian, while for Wroclaw they relied on the survey performed by the implementing body itself.

Second visits to the cities

With a clear view emerging from desk research and project survey as to the overall relevance, efficiency and effectiveness of each ECOC, the contractor undertook a second three day visit to each city. The visit focused on conducting in-depth interviews with the managing teams, their key stakeholders and cultural operators identified during the initial visits- please see the annex for more information.

Building on the interviews undertaken in the first visits, the interviews with the delivery teams served four purposes: i) gathering further factual data and information about the cultural programme and its achievements; ii) identifying the "story" of the ECOC throughout its lifecycle, i.e. conception, application, development, delivery, legacy; iii) gaining a critical (albeit "insider") perspective on the relevance, efficiency, effectiveness and sustainability of the ECOC; iv) triangulating emerging findings, for example, those emerging from the desk research and project survey.

Interviews with other key stakeholders in the city (e.g. municipality, chamber of commerce, commercial sponsors, media representatives, and cultural operators) were also essential to identifying the overall "story" of the ECOC from conception, through application, to development and delivery. These individuals offered an external, alternative and broader perspective on the ECOC.

Additional project and stakeholder interviews

These interviews sought to gather an in-depth understanding of the effect that ECOC had for the individuals involved, their organizations and the local culture sector as a whole. It also allowed to ensure that findings are based on consensus across a range of target groups as well as to identify key differences in the experience of different stakeholders. In the case of projects, the criteria for inclusion focused on whether it represents good practice in relation to at least one of the key dimensions of the evaluation, including (but not limited to) increased European cooperation, the effective targeting of key groups of citizens or neighbourhoods, audience development strategies, or legacy effects.

Analysis and reporting phase

Once all the information gathered, the experts fulfilled an analysis of the available information, drawing conclusions and triangulating data to ensure consistency and accuracy. The limitations of the data gathered have been explained in the report and in the Staff Working Document. After the first drafting exercise, the results were shared with the two ECOC for a factual check. Both the Donostia San Sebastian and Wroclaw ECOC were asked to undertake a check on the completeness of data being used to address each evaluation topic. Their comments, where relevant, were included in the Final Report submitted by the contractor supporting the evaluation.

(1)      Decision No 1622/2006/EC of the European Parliament and of the Council of 24 October 2006 establishing a Community Action for the European Capital of Culture event for the years 2007 to 2019 (OJ L 304, 3.11.2006, p. 1), available at: - http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=OJ:L:2006:304:0001:0006:EN:PDF .

(2)      Decision No 445/2014/EU of the European Parliament and of the Council of 16 April 2014 establishing a Union action for the European Capitals of Culture for the years 2020 to 2033 and repealing Decision No 1622/2006/EC (OJ L 132, 3.5. 2014, p. 1), available at:    - http://eur-lex.europa.eu/legal-content/EN/TXT/?uri=uriserv:OJ.L_.2014.132.01.0001.01.ENG This Decision, which covers the ECOC titles 2020 to 2033, retains the general structure and main elements of the previous Decision while introducing improvements to maximise the benefits of holding the title as well as taking part in the competitive process for all bidding cities and their citizens. Improvements include among others the introduction of more explicit and measureable criteria, the reinforcement of conditionality for the payment of the Melina Mercouri Prize and the obligation for the cities – instead of the Commission – to carry out the ex-post evaluation of the ECOC year.

(3)      Available at: https://ec.europa.eu/programmes/creative-europe/sites/creative-europe/files/library/2011-capitals-culture-assignment-report_en.pdf

(4)      Article 17 of Decision No 445/2014/EU.

(5)      Full document available at: https://ec.europa.eu/culture/evaluations_en .  

(6)      Resolution of the Ministers responsible for Cultural Affairs, meeting within the Council, of 13 June 1985 concerning the annual event 'European City of Culture' (85/C 153/02), available at;- http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=OJ:C:1985:153:0002:0003:EN:PDF .

(7)      Decision 1419/1999/EC of the European Parliament and of the Council of 25 May 1999 establishing a Community action for the European Capital of Culture event for the years 2005 to 2019 (OJ L 166, 1.7.1999, p. 1). The Decision was amended by Decision No 649/2005/EC to integrate the Member States that joined the EU in 2004 (OJ L 117, 4.5.2005, p. 20). Both Decisions are available at:- http://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A31999D1419 - http://eur-lex.europa.eu/eli/dec/2005/649(1)/oj

(8)      Decision (EU) 2017/1545 of the European Parliament and of the Council of 13 September 2017 amending Decision No 445/2014/EU establishing a Union action for the European Capitals of Culture for the years 2020 to 2033 (OJ L 237, 15.9.2017, p. 1). Text available at: - http://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32017D1545  

(9)      Regulation (EU) No 1295/2013 of the European Parliament and of the Council of 11 December 2013 establishing the Creative Europe Programme (2014 to 2020) and repealing Decisions No 1718/2006/EC, No 1855/2006/EC and No 1041/2009/EC (OJ L 347, 20.12.2013, p.221).

(10)      Selection of the European Capital of Culture for 2016 in Spain, Final selection17/28 June 2011, p.5, available at: https://ec.europa.eu/programmes/creative-europe/sites/creative-europe/files/files/ECOC-2016-panel-spain_en.pdf . 

(11)      All the detailed long-term goals of Wroclaw2016 are available at: www.wroclaw2016.pl/about-ECOC .

(12)      See previous evaluation reports available at: https://ec.europa.eu/programmes/creative-europe/actions/capitals-culture_en  (table at the bottom of the document).

(13)      A total of 29 stakeholders for Donostia San Sebastian and 30 for Wroclaw were interviewed.

(14)      29 projects responded to the survey in Wroclaw while 52 projects responded to the project survey carried out by the Donostia San Sebastian 2016 Foundation.

(15)      Available at: http://ec.europa.eu/smart-regulation/roadmaps/docs/2016_eac_015_evaluation_ecoc_2016.pdf .

(16)      The budget allocated to the evaluation work (approximately 70 000 €) is proportionate to the low level of EU funding directly provided to the ECOC (i.e. an award based on a recommendation of the panel after the final monitoring meeting in the form of the €1.5m Melina Mercouri Prize).

(17)      Decision No 1622/2006/EC requires that the Commission conducts the evaluation immediately after the title year.

(18)      See footnote 5.

(19)      Available at: http://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A12008E167 .

(20)      During the Second World War, Jewish and Polish citizens were persecuted by the Third Reich, although the city largely escaped damage until the final few months of the war. The advancing Soviet armies surrounded the city during the three-month-long “Battle of Breslau” which culminated in considerable destruction, loss of life and the eventual surrender by German forces on 6 May 1945. After the war, the city became part of Poland under the terms of the Potsdam Agreement. Most of the remaining German population of 190,000 fled or were forcibly removed, to be replaced by ethnic Poles, most of whom were resettled from parts of eastern Poland that were incorporated into the Soviet Union.

(21)       http://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:02012R0966-20160101&from=EN

(22)       http://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:02012R1268-20160101&from=EN

(23) http://ec.europa.eu/smart-regulation/roadmaps/docs/2016_eac_015_evaluation_ecoc_2016.pdf

(24)       https://ec.europa.eu/programmes/creative-europe/sites/creative-europe/files/files/ecoc-2016-evaluation-en_0.pdf

(25)      See previous footnote.