University of Sussex

File(s) under permanent embargo

Reliability study of the European appropriateness evaluation protocol

journal contribution
posted on 2023-06-07, 14:48 authored by S Lorenzo, T Lang, R Pastor, A Tampieri, B Santos-Eggimann, H Smith, A Liberat, J Restuccia
Objective: To help to co-ordinate and harmonize research on utilization review in Europe, the US Appropriateness Evaluation Protocol (AEP) was adapted for use in the European setting. The aim of this paper is to assess the reliability of the European version of the AEP (EU-AEP). Design: Nineteen English-language medical records were reviewed by a physician reviewer from each of six participating countries: Austria, France, Italy, Spain, Switzerland and the UK. Each of the six reviewers was asked to assess the appropriateness of the 19 admissions and 31 hospitalization days (19 admission days and 12 randomly selected days of hospital stay, excluding days of discharge) using the revised review instrument. To evaluate inter-rater reliability, the ? statistic was used to measure overall and pair-wise agreement for the assessment of appropriateness of admission and of day of care, respectively. Results: For admission, the overall ? statistic among the six reviewers was 0.64, with ? values for each pair of reviewers in the range 0.46-0.86. For day of care, the ? was 0.59, with pair-wise ?coefficients in the range 0.25-0.95. Conclusion: The observed agreement could be considered substantial, especially if the fact that medical records were handwritten in a language native to only one of the reviewers is considered. Besides all the study limitations, this finding provides at least preliminary support for the application of the EU-AEP as a reliable instrument in the European setting, including application in comparative studies involving two or more countries.


Publication status

  • Published

File Version

  • Published version


International Journal for Quality in Health Care




Oxford University Press





Page range


Department affiliated with

  • Primary Care and Public Health Publications

Full text available

  • No

Peer reviewed?

  • Yes

Legacy Posted Date


First Compliant Deposit (FCD) Date


Usage metrics

    University of Sussex (Publications)


    No categories selected