Reliability study of the European appropriateness evaluation protocol

Objective: To help to co-ordinate and harmonize research on utilization review in Europe, the US Appropriateness Evaluation Protocol (AEP) was adapted for use in the European setting. The aim of this paper is to assess the reliability of the European version of the AEP (EU-AEP). Design: Nineteen English-language medical records were reviewed by a physician reviewer from each of six participating countries: Austria, France, Italy, Spain, Switzerland and the UK. Each of the six reviewers was asked to assess the appropriateness of the 19 admissions and 31 hospitalization days (19 admission days and 12 randomly selected days of hospital stay, excluding days of discharge) using the revised review instrument. To evaluate inter-rater reliability, the ? statistic was used to measure overall and pair-wise agreement for the assessment of appropriateness of admission and of day of care, respectively. Results: For admission, the overall ? statistic among the six reviewers was 0.64, with ? values for each pair of reviewers in the range 0.46-0.86. For day of care, the ? was 0.59, with pair-wise ?coefficients in the range 0.25-0.95. Conclusion: The observed agreement could be considered substantial, especially if the fact that medical records were handwritten in a language native to only one of the reviewers is considered. Besides all the study limitations, this finding provides at least preliminary support for the application of the EU-AEP as a reliable instrument in the European setting, including application in comparative studies involving two or more countries.