4.8.1 Referral quality
The aim of the implementation of the referral template was to increase the quality of referrals and then to assess whether this translated into improved care. Hence, a measure of referral quality was paramount to the completion of this study. The scoring system was based on the referral template used, with one point awarded for each clinical detail
34
requested in the referral template that was present in the received referral. This is similar to scoring systems used in other referral evaluations[34,38]. No weighting was applied.
Referrals were scored by three raters, blinded to the intervention status of the patient. However, when the electronic template was used the intervention status was sometimes obvious. Twenty percent of the referrals were scored independently by two raters, no referral was scored by all three raters, and the three rater-pairs shared at least 25 referrals each.
4.8.2 Patient experience
Patient experience is an ever more important aspect in the evaluation of health care interventions, and the evaluation of patient experience can help drive improvements in quality of care[130]. Better patient experiences are associated with safety and clinical
effectiveness[131]. Multiple tools exist to measure various aspects of care coordination[132]
and patient experience. However, after a thorough search, no relevant questionnaire was found in Norwegian. Therefore, a patient experience questionnaire was developed for the current project; it aimed to measure patient experience with the care coordination and treatment process. The questionnaire included all questions from the full version of the Generic Short Patient Experience Questionnaire[133,134] and the two questions about health interaction from the Commonwealth Fund Survey 2010[135]. Further questions were added to assess (1) who referred the patient, (2) if the referral was seen as appropriate, and (3) an overall assessment of the hospital (Table 3). The full questionnaire is included in the Appendix. The questionnaire was piloted with local health professionals and patients to evaluate the content, face validity, and acceptability[136,137]; however, no further formal evaluation of the questionnaire was done.
Table 3 – Questionnaire details Question
no
Item text
1 Did the cliniciansa talk to you in a way that was easy to understand?
2 Do you have confidence in the clinicians’ professional skills?
35
3 Did you get sufficient information about how examinations and tests were to be performed?
4 Did you get sufficient information about your diagnosis/conditions?
5 Did you perceive the treatment to be adapted to you situation?
6 Were you involved in decisions regarding your treatment?
7 Did you perceive the institution work practices as well organised?
8 Did you perceive the equipment at the institution to be in good working order?
9 Overall, was the help and treatment you received at the institution satisfactory?
10 Do you believe that you were in any way given incorrect treatment (according to your own judgement)?
11 Did you have to wait before you were given an appointment at the institution?
12 Overall, what benefit have you had from the care at the institution?
13 Did the hospital specialist lack basic medical information from your GP about the reason for your visit or test results?
14 After your saw the hospital specialist, did your GP lack important information about the care you got from the specialist?
15 Was the referral to the out-patient department necessary (according to your own judgement)?
16a Were you referred by your GP for the out-patient appointment?
16b If no in question 16a; who referred you?
17 If you take an overview of your entire treatment process, how would you evaluate the institution?
a ‘Clinicians’ refers to those who had the main treatment responsibility. This is linguistically clearer in the Norwegian wording.
The questionnaire was mailed to study participants after their hospital appointment.
To increase the response rate, pre-paid envelopes were provided, addresses were hand-written, the questionnaire was kept as short as possible, and association with the research body was clearly indicated[138]. For initial non-responders, a mail reminder was sent approximately 1 month after the first questionnaire was sent out.
36 4.8.3 Health process outcomes
To further assess the effect of the referral template on the care pathway, information on other health process outcomes was retrieved through a manual review of EHR:
wait time from date of referral to date of first hospital appointment in days
time from referral to initiation of treatment in days
number of hospital appointments before a diagnosis was made
outcome of referral – hospital appointment/return information/referral rejected
application of ‘right to health care’
‘Right to health care’ is a legal term in Norwegian health care[3]. Every referral received is evaluated by a hospital doctor, who determines whether that individual patient, in that care pathway, has the right to prioritised health care. The ‘right to health care’ is applied only if the patient can be expected to benefit from the health care process and if the cost is in accordance with the expected benefit. Patients to whom this legal right is applied are then assigned a maximum wait time in accordance with prioritisation
guidelines[110,139,140].
In addition, the positive predictive value was calculated as it was for glue ear referrals in otolaryngology by Bennett et al[141]. When adapting this concept to a hospital medical out-patient department, we defined positive predictive value as the proportion of referrals that led to a histological diagnosis, diagnostic clarification, or change in medical
management. This was scored by the scoring panel on the same scoring sheets as the quality of care criteria, though under a separate heading.
The scoring panel also applied a subjective assessment of the care pathway. This was done in two ways. Firstly, a quality rating of the treatment process was given on an ordinal scale of 1-10. Secondly, the scorers assessed whether the treatment process was
appropriate with a binary yes/no response.
To evaluate the correlation between seriousness of diagnosis and initial prioritisation, the variable ‘wait time’ was correlated with severity of final diagnosis. Wait time was defined as the time from the referral was received at the hospital until the first out-patient
appointment, measured in days. To differentiate between the seriousness of the various
37
illnesses, the possible outcome diagnoses were grouped into two, three, or four severity levels. The four-level structure was used for the analysis in Paper III. The categorisation was done by the PhD candidate based on the International Classification of Diseases tenth revision (ICD-10) diagnostic codes, and it was reviewed by specialists in the relevant field. If more than one diagnosis was located for an individual patient, the ICD-10 code associated with the highest severity grouping was used in further analyses.