52003DC0086

REPORT FROM THE COMMISSION TO THE COUNCIL on the implementation of the Eurofarm project (in accordance with Article 13 of Council Regulation (EEC) No 571/88)

Introduction

Council Regulation (EEC) No 571/88 [1], on the organisation of Community surveys on the structure of agricultural holdings, hereafter referred to as the "basic Regulation", defines the Eurofarm project as a "set of data banks to be used for processing Community surveys on the structure of agricultural holdings for the requirements of national and Community agricultural policies".

[1] OJ L 56 of 2.3.1988, p. 1, last amended by Commission Regulation (EC) No 143/2002 (OJ L 24 of 26.1.2002, p. 16).

Article 13 of the basic Regulation states that every three years the Commission shall submit to the Council a report on the implementation of the Eurofarm project and shall also propose any necessary changes to the Regulation.

The last report on the implementation of the Eurofarm project was adopted by the Commission on 23 April 1999 [2] and considered by the Council's Special Agricultural Committee on 14 February 2000. This is the fourth report to be submitted by the Commission to the Council.

[2] COM(1999) 189 final.

The current report presents in particular:

- an overview of the activities in connection with the 1997 survey

- a progress report on the activities in connection with the 1999-2000 basic survey

- a short description of the preparatory work for the 2003 survey

- a progress report on data processing developments concerning the Eurofarm system.

A. 1997 survey

The results of the 1997 survey (tabular data for Germany - individual data for the other Member States) were transmitted to Eurostat in accordance with the agreed definitions and formats. The 1997 results were analysed and published to a greater and more detailed extent than the results of the previous surveys. [3]

[3] Eurostat publications:

A great deal of aggregated data are now included in the Eurostat databases. Nevertheless, the following limiting factors should be highlighted as far as the potential for exploitation of the results are concerned:

- the data sometimes lack freshness because of their delayed transmission by some Member States and the slowness of the validation process;

- Germany did not provide microdata;

- users have easy access only to predefined tables;

- responses to ad hoc requests are hampered by cumbersome procedure.

The Community contribution to the 1997 survey amounted to 13.8 Mio euro in total.

B. 1999-2000 basic survey

In order to establish the implementing procedures for the 1999-2000 survey and to adapt the typology of agricultural holdings, the Commission adopted five decisions [4] in line with the procedure laid down in Article 15 of the basic Regulation.

[4] Commission Decision 1998/377/EC of 18 May 1998 adapting Annex I to Council Regulation (EEC) No 571/88 in view of the organisation of the Community surveys on the structure of agricultural holdings (OJ L 168 of 13.6.1998).

The list of characteristics surveyed was in fact adapted to take account of new information requirements in connection with rural development and methods of sustainable production. The importance of the survey on the structure of agricultural holdings for the production of agri-environmental indicators was emphasised in the communication from the Commission to the Council and the European Parliament on 20 March 2001 [5]. Unfortunately, the optional variables (relatively numerous) were supplied by only some of the Member States, thus making it impossible to compile the corresponding Community totals.

[5] Communication from the Commission to the Council and the European Parliament "Statistical Information needed for Indicators to monitor the Integration of environmental concerns into the Common Agricultural Policy", COM(2001) 144.

The 1999-2000 surveys were conducted in all the Member States between 1.12.1998 and 1.3.2001 and covered 6.8 million holdings. The individual data were transmitted to Eurostat by all the Member States (including Germany, for the first time). They were transmitted between August 2000 and July 2002 (the individual countries' transmission dates are shown in Annex). One point to note is that the deadlines laid down by Decision 1999/714/EC were respected with regard to the initial transmissions of non-validated data by all Member States except Ireland (which transmitted its results in July 2002 instead of December 2001). The validation of the data was sometimes spread over several months, but by August 2002 all the validated individual data were in the Eurofarm individual data base apart from the data for Ireland (October) and Italy (November).

Experience has shown that the production of all the tables to be stored in the Eurofarm tabular data base in accordance with Commission Decision No. 1999/799/CE is cumbersome and time-consuming. In order to reduce the delays required to make results available to users;

- a list of simplified and better structured tables was devised in agreement with the Member States; the corresponding tables will gradually become available in Eurostat's NewCronos data base;

- the computer programs will be optimized.

Several Eurostat reports mainly based at the 1999-2000 Survey will be widely disseminated concerning in particular

- the main results of the 1999-2000 Survey

- the age of farmers

- the female employment in agriculture

- organing farming

- grazing livestock

- pigmeat industry (environmental impact)

It should be easier to use the results of the 1999-2000 survey thanks to the new table generator developed as part of the support for management of Eurofarm, and thanks to a speedier procedure for validating ad hoc tables (simultaneous transmission of ad hoc tables to users and statistical services in the event of urgent requests).

The payment of the Community contribution to the 1999-2000 survey (including, where appropriate, an additional contribution of 50% for a complete survey) began in 2000 and should be completed in 2003. In total the Community contribution should amount to some 26.6 Mio euro.

C. 2003 survey

With a view to the 2003 survey, the Commission has adopted three regulations [6] in accordance with the procedure laid down in Article 15 of the basic Regulation. The number of optional variables has been restricted (in order to avoid complicating and overburdening the management of data without the possibility of compiling Community totals), and a section focussing on environmental aspects of agriculture was added to the list of characteristics to be collected for this survey. An important development for this survey is that the transmission deadlines now refer to the transmission of already validated data (so that delays in validation should not affect the date foreseen for availability of results).

[6] Commission Regulation (EC) No 143/2002 of 24 January 2002 amending Annex I to Council Regulation (EEC) No 571/88 in view of the organisation of the Community surveys on the structure of agricultural holdings in 2003, 2005 and 2007 (OJ L 24 of 26.1.2002).

In order to speed up the availability of the main results and to avoid excessive time lags between the dissemination of the data by the Member States and the Commission, a common set of tables would be transmitted to Eurostat by the Member States as soon as the initial national data were disseminated.

D. Data processing developments concerning the Eurofarm system

D.1 Software developments 1997-2002

D.1.1 Introduction

As envisaged in the last Report to the Council on the implementation of the Eurofarm project, major developments in the Eurofarm software have been conducted. Certain of these became necessary as the hardware and operating system used for the 'old' Eurofarm system are no longer supported by the Commission's Directorate of Informatics.

However, the migration of Eurofarm to a new computer system has also provided the opportunity to improve the functionality of the Eurofarm system and to find solutions for certain imperfections identified in the old system.

D.1.2 Migration of Eurofarm to a UNIX platform

Transfer of the Eurofarm database to the new system was carried out in a step-wise manner, which allowed to set-up all (meta-) data for the migration of 'old' data from the mainframe and to process incoming data from the 2000 survey.

The software for the new Eurofarm system was developed by an external company, using market-standard software tools. Data are stored in Oracle databases running on a UNIX server and are accessed from PCs equipped with software developed with Oracle tools. UNIX and Oracle are software tools widely used in- and outside the Commission. By using these tools, the Eurofarm system has been brought into line with the Commission's informatics policy. This had not been the case with the previous system where for different reasons a wide range of software tools were used, some of which were out-dated (Cobol, MVS) or were niche products without a clear future (Acumen).

Special attention has been made to facilitate the management of the Eurofarm system and to bring the management requirements into line with the Commission's personnel policy. While the software development has been made by an external company, all processing of the farm structure data can be made by non-informatics personnel having an appropriate understanding of the application logic. This and consequent personnel decisions made it possible to dedicate all data processing to Eurostat officials without recourse to external personnel.

Since spring 2002 ad-hoc tables from the 2000 survey have been created with the new system to respond to user demands and since October 2002 tables created with the new system are progressively being published in Newcronos.

The total development costs for the new Eurofarm system were 900,000 euro, part of which was financed by the Commission's Informatics budget and the rest by the budget for the farm structure survey.

D.1.3 New Structure of the Eurofarm Database System

According to their function, the data relevant for farm structure survey processing have been separated into three databases:

- "metadatabase"

This database stores "metadata" needed for the processing of the Eurofarm data (from the definition of the survey files through to the validation of incoming data and the selection of the parameters needed to define statistical tables).

Information stored in this database (such as the list of survey characteristics, typology rules etc.) covers all surveys from the basic 1989/1990 survey onwards.

The approach to develop a "metadatabase" has been chosen in order to centralise all types of parameters previously distributed over different files in one central repository to make them easily accessible and the processing of the Eurofarm data transparent to the end-users, as well as to facilitate expected future extensions (future surveys - candidate and EFTA countries).

- "individual database"

This database allows a consistent view of all survey data from the different surveys and countries.

Special attention has been paid to develop all processing programs in a uniform manner for the different survey years and different countries. Specific differences for the different survey years (list of characteristics etc.) are managed via the metadatabase and not -as in the old system- hidden in the programs themselves.

The database currently contains a total of 15 439 476 records.

- "tabular database"

This database contains the aggregated tables which are derived from the individual data of the "individual database" using the table definitions stored in the "metadatabase".

In order to ensure that the processing requirements meet the necessary confidentiality restrictions, this database has been separated into a confidential part containing the raw tables and a "public" part containing filtered tables. The latter are derived by taking into account the confidentiality requirements not to make available any data referring to individual entities.

Table generation is now based on one single table generator which uses variable and dimension definitions stored in the metadatabase. This is seen as a major improvement in comparison with the previous system where for different groups of tables specific programs were developed. This new functionality gives the flexibility to respond to new demands for tables without the need for writing new programs.

The tabular database is used as central storage of all tabular data. Table data from this database are extracted to respond to ad-hoc requests, to copy them into Eurostat's general dissemination database or to provide tables for paper publications.

D.2 Planned developments 2003-2004

Eurostat is currently planning three projects to improve the functionality of the system so far developed and make the farm structure survey results more accessible to users:

(1) Providing access to a centralised validation system for data providers (Member States, Candidate Countries, EFTA countries)

The validation process from the delivery of the first data-set until the acceptance of the final version often takes several months. In order to speed up this process and to make validated farm structure data accessible sooner, it is intended to set up a system within the Commission's data centre which can be accessed by data providers. This system will allow data providers to validate data themselves without the need to wait for a response from Eurostat.

(2) Facilitate the access to and the performance of the table generator

While the new table generator has the core functionality required, additional efforts will be undertaken to make the functionality to create statistical tables more widely available. The current version requires installation of programs and direct connection to Eurostat's network. It is planned to develop a system which will allow the creation of tables using a standard internet access to the Eurofarm (sub-)system. This will allow appropriate users working outside Eurostat's premises, and in particular relevant national statistical bodies, to create tables to meet their specific requirements for analysis. In the context of this project, features made available in the latest Oracle software will be used to improve the overall performance of the system.

(3) Providing access to anonymised individual data

The Eurofarm system produces multidimensional tables based on the farm structure data. This is not in all cases the best way to present statistical information. Studies based on multivariate analysis need different types of information (e.g. the distribution of certain characteristics or correlations between different characteristics).

Eurostat is currently looking at the different aspects related to the dissemination of anonymised individual data. Different methods to derive from the confidential individual data modified 'individual' data which can be disseminated have to be examined, so that users of the data may have the flexibility to derive statistics in forms other than tabular data.

ANNEX

Farm structure survey 2000 -Transmission and validation dates

>TABLE POSITION>

In average the total delay was 253 days, of which 70 days were required in Eurostat and 183 days were required in Member States