ANNEX

Quality criteria and statistical concepts

The quality report should contain quality-related data and metadata in line with the following quality criteria and statistical concepts.

If a particular statistical concept is not relevant to a statistical operation, that concept should remain in the quality report, accompanied by the words ‘Not applicable’.

1.   CONTACTS

Individual or organisational contact points for data or metadata, including contact details.

2.   STATISTICAL PRESENTATION

Description of the data disseminated, which can be displayed to users as tables, graphs or maps.

2.1.   Data description

Description of the main characteristics of the dataset.

2.2.   Classification systems

Where applicable, the list of classifications and breakdowns used in data, and any deviations from European statistical standards or international standards.

2.3.   Sector coverage

Description of the main themes covered by the dataset.

2.4.   Statistical concepts and definitions, including the reference period

The list of all the variables that deviate from the standard definition, mentioning the national concepts used and any differences between national concepts and the respective data collections.

2.5.   Statistical units

Description of the observation units.

2.6.   Statistical population

Description of the target statistical population or populations to which the dataset refers, i.e. the population about which information is to be collected.

2.6.1.   Population(s) not covered

Information about any sub-populations not covered by the data collection (e.g. homeless people or people living in institutions), including description of any such population and its best quantitative estimate.

2.7.   Reference area

Description of the geographical area to which the statistical phenomenon measured relates: the geographical area covered and a list of any regions that are excluded.

2.8.   Time coverage

The periods or points in time to which the observation refers.

3.   STATISTICAL PROCESSING

Operations performed on data to derive new information in accordance with a given set of rules.

3.1.   Source data

Description of the source of the raw statistical data (e.g. interviews, administrative data, any other sources). If administrative registers are used, they should be clearly described (source, primary purpose, possible shortcomings, etc.)

3.1.1.   Sampling frame

Description of the methods used to obtain or create the sampling frame.

3.1.2.   Sample design

Description of the following aspects:

—	Type of sampling design (stratified, multi-stage, clustered, one stage, two stages)

—	Stratification and sub-stratification criteria

—	Sample size

3.2.   Frequency of data collection

Information on frequency at which a dataset is collected.

3.3.   Data collection

Description of the methods used to gather data (CAPI, CAWI, CATI, etc.). The national questionnaire used for data collection should be attached, along with its translation into English.

3.4.   Data validation

Description of procedures used for checking and validating the source and output data, including explanation of how the results of these validations are monitored and used.

3.5.   Data compilation

Description of the data compilation process (e.g. data editing, imputation, weighting, adjustment for non-response, calibration, model used, etc.) Each step of weighting should be described separately: calculation of design weights; non-response adjustment (how the design weight is corrected, taking account of differences in response rates); calibration (the level and variables used in the adjustment, method applied); calculation of final weights.

4.   QUALITY MANAGEMENT

Systems and frameworks in place within an organisation to manage the quality of statistical products and processes.

4.1.   Quality assurance

Description of the quality assurance framework and/or quality management system (e.g. EFQM, ISO 9000) used in the organisation.

4.2.   Quality assessment

Description of the overall quality of statistical outputs, summarising the main strengths and any quality deficiencies in the standard quality criteria: relevance, accuracy, reliability, timeliness, punctuality, comparability and coherence. Any trade-offs between quality aspects and any quality improvements foreseen can be mentioned.

5.   RELEVANCE

5.1.   User needs

Information (if available) on (any new) needs of the users in relation to data collected.

5.2.   User satisfaction

Information (if available) on level of satisfaction of data users in relation to data collected and made available.

5.3.   Completeness

Description of any non-compliance in terms of variables that are not transmitted.

6.   ACCURACY AND RELIABILITY

6.1.   Overall accuracy

Summary of the various components of an assessment of accuracy, linked to a given dataset or domain:

—	Description of the main sources of random and systematic errors in the statistical outputs, with a summary assessment of all errors, focusing particularly on the impact on key estimates.

—	If relevant, data revision aspects.

6.2.   Sampling error

—	Description of the methodology for calculating precision estimates.

—	Precision measures of estimates in accordance with the technical specifications of the individual datasets.

—	The standard errors at national and, where required, regional level (NUTS 2) for the main indicators as mentioned in Annex II to Regulation (EU) 2019/1700.

6.3.   Non-sampling error

6.3.1.   Coverage error

Description of divergence between the frame population and the target population.

—	Frequency and timing of frame updates.

—	Errors due to discrepancies between the sampling frame and the target population and sub-populations (over-coverage, under-coverage, misclassifications).

6.3.2.   Measurement error

Description of errors that occur during data collection, causing recorded values of variables to differ from true values.

—	Description of efforts made in questionnaire design and testing (including addressing errors arising from multi-mode and/or multi-source data collection).

—	Description of interviewer training.

—	Proxy interview rates.

6.3.3.   Non-response error

Description of:

—	Available characteristics of non-respondents.

—	Unit and item non-response rates.

—	Substitution rates.

—	Gross sample size (initial sample size), number of eligible units and net sample size, including substitution units (achieved sample size)

6.3.4.   Processing error

Description of any error in processing and its impact on final data collection results, arising from faulty implementation of correctly planned implementation methods.

—	Description of quality checks and the data editing process.

—	Description of imputation procedures.

—	Imputation rates.

6.3.5.   Model assumption error

Where applicable: description of error arising from domain-specific models needed to define the target of estimation.

6.4.   Seasonal adjustment (where applicable)

Description of statistical techniques used to remove seasonal effects that influence a data series.

6.5.   Data revision – policy

Description of policy aimed at ensuring the transparency of disseminated data, with preliminary data being revised after compilation.

6.6.   Data revision – practice

Information on data revision practice.

7.   TIMELINESS AND PUNCTUALITY

Information on:

—	Date of the dissemination of national results.

—	Number of days between the end of fieldwork and the first fully validated delivery of data to the Commission (Eurostat).

—	Date of the first full delivery of data to the Commission (Eurostat). If the delivery of data does not meet the deadline laid down in Regulation (EU) 2019/1700, a reason for the delay should be given.

8.   COHERENCE AND COMPARABILITY

Description of how the requirements established in the specific domain have been fulfilled, including, where relevant, the impact of any deviations from the questionnaire and definitions.

8.1.   Comparability – geographical

Description of any problems of comparability between different regions of the country.

8.2.   Comparability – over time

Information on the length of comparable time series, including the years when any series breaks occurred, and the reasons for them.

8.3.   Coherence – cross domain

Comparison with external sources for all relevant variables, where the Member States concerned consider such external data to be sufficiently reliable.

8.4.   Coherence – sub-annual and annual statistics

If applicable.

8.5.   Coherence – National accounts

If applicable.

8.6.   Coherence – internal

Information on any lack of coherence in the output of the statistical process.

9.   ACCESSIBILITY AND CLARITY

Information on:

—	Dissemination formats.

—	Documentation on methodology and quality.

10.   COST AND BURDEN

Burden on respondents and, if available, cost associated with the collection and production of the statistical product. The average duration of household interviews should be stated. If possible and relevant, the duration of household interview should be reported by mode of data collection.

11.   CONFIDENTIALITY

Information on ownership of data, indicating the extent to which their unauthorised disclosure could be prejudicial or harmful to the interests of the source or other relevant parties.

—	Confidentiality policy - description of any provisions in addition to European legislation that are relevant to the statistical confidentiality applied to the data.

—	Confidentiality – data treatment: general description of the rules applied to treating microdata and macrodata (including tabular data) with regard to statistical confidentiality.

12.   COMMENT

Supplementary descriptive text that can be included in the quality report.